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United States 

General Accoonting Office 

Washington, D.C. 20S48 



Program Evaluation and 
Methodology Division 

B-228638 

August 4, 1988 

The Honorable Wiiiiain D. Ford 

Chairman, Ck)inmittee on Post Office and Civil Service 

House of Representatives 

Dear Mr. Chairman: 

At your request, we have examined the feasibility of assessing the quality of the federal 
civilian workforce. As was agreed, we focused on professional and administrative staff. Our 
conclusion is that such an assessment can be made. 

In order to reach this conclusion, we developec* a measurable definition of quality centered 
on attributes of the individual and the match of the individual's capabilities to the needs of 
the job. We examined agency personnel data to see whether indicators of quality pertinent to 
this definition were readily available and found that they were not. We then developed 
proposals for new data-gathering that would permit policymakers and others to obtain 
answers to specific common questions about the quality of those entering and leaving the 
professional and administrative workforce, and about changes in overall quality across time. 

As arranged with your office, unless you publicly announce the contents of this report 
earlier, we plan no further distribution of it until 30 days from the date of the report. At that 
time we will send copies to the Of Hce of Personnel Management and others who are 
interested and make copies available to others upon request. 

Sincerely yours. 




Eleanor Chelimsky 
Director 



Executive Suiranary 



The federal workforce costs taxpayers billions of dollars annually and 
affects all of our lives through the performance of its functions. A gen- 
eration or more ago, so beliefs run, public service was a high calling, 
attracting the best and the brightest. While such a golden age perhaps 
never existed, it is clear that in recent years concern has intensified 
about changes in the attractiveness of public service and possible conse- 
quences for the quality of the federal workforce. 

The House Committee on Post OfHce and Civil Service asked gao to 
(i) to examine the feasibility of producing generalizable and reliable 
information on the quality of the federal workforce over time and (2) in 
the event this was feasible, to outline a design for obtaining it. 



The overall federal civilian workforce (including legislative, judicial, and 
executive branch emfdoyees and employees of the U.S. Postal Service) 
includes over 3 million people nationwide, about 3 percent of all civilian 
employees in the United States. Of the non^postal workforce of 1.86 mil- 
lion, 42 percent is concentrated in professional and administrative posi* 
tions such as accountant, attorney, engineer, or contract specialist. gao*s 
review addresses the executive branch civilian workforce in profes- 
sional and administrative positions. 

The quality of the federal workforce is important because it presumably 
affects performance: that is, government agencies' ability to carry out 
their responsibilities effectively and economically. Direct measures of 
the performance of some govemmafit functions are possible, as shown in 
productivity data reported by the Bi^reau of Labor Statistics. Trends in 
such data, however, can reflect a number of different influences includ* 
ing increased automation, improved teleconununications, or manage- 
ment reforms, along with changes in the workforce. Thus, data on 
performance could not address the issue of concern to the conunittee: 
the effectiveness ot government efforts to recruit, develop, and retain a 
workforce with the quality needed for efficient performance of govern- 
ment functions. 



Results in Brief purpose of selecting indicators, gao developed a two-part defini- 

tion of quality including both employee capabilities and the degree to 
which those caDabiUties are matched to the requirements of a particular 
job. Judging whether employees in a particular occupation are of high 
quality thus involves not only an absolute level of knowledge, skill, and 
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ability, but also congruence between those capabilities and the needs of 
the work. (See chapter 2.) 

GAG found that the information currently available in agency personnel 
records was not comprehensive, accessible, or current enough for the 
purposes of the quality assessment effort, gag also found that it would 
not be practical to change agency systems so that they might provide a 
better data base for the purpose in the future. (See chapter 3.) 

GAG recommends an alternative approach, to be implemented by the 
executive branch, based on new d^ to be gathered chiefly by survey- 
ing samfdes of employees in selected occupations. The proposed data 
would permit generalized answers to many of the msyor questions that 
have been raised about quality in the three different segments of the 
workforce. (See chapters 4-6.) The approach s^pears to be relatively 
low in direct federal costs (see appendix III), and it has the potential to 
be refmed and expanded over time as experience accumulates on its 
strengths and limitations. (See chaptpr 7.) 



Q^Q'g Analysis ^ definition of quality includes two basic concepts: the capabilities 

^ of the individual and the match between the extent of those capabilities 

and the needs of the specific job. To define and measure an individual's 
capabilities, gag selected two basic characteristics: knowledge, skill and 
ability; and attitudes, values, and motivation. There are many types of 
data which can be used to show each of these, including education, 
training, test scores and licenses as indicators of knowledge, skill, and 
ability, and individuals' views as indicators of attitudes, values, and 
motivation. 

gag believes an indicator of the match between individuals' capabilities 
and their current work should be obtained, at least initially, from rat- 
ings by both individual employees and their supervisors. (See chapter 
2.) 



Elxisting Personnel Data After reviewing the data maintained on employees in all forms at all 

levels in eight agencies, gag concluded that only a few aspects of its 
deflnition of quality could be measured from existing records. Auto* 
mated Hies contain a few indicators pertinent to workforce quality, 
chiefly on education. These are, however, not up to date. Apart from 
scattered surveys of small groups, no data exist that could shed light on 
other parts of the definition of quality such as attitudes, values and 
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motivation or the match of capabilities to the needs of the job. Some 
relevant information might be found in of Hdal personnel files, but 
would be costly to retrieve. There appear to be signiHcant barriers to 
expanding agency personnel data systems to add routine collection and 
storage of additional data on gao*s deHnition of quality. The burden of 
data-collt:ction and data-entry would be too great and the motivation 
within the agencies to achieve completeness and accuracy too low, to 
yield worthwhile information. (See chapter 3.) 



The xaajor questions that an adequate assessment of workforce quality 
must answer are: How good are those being attracted to government 
jobs and how do they compare to those taking jobs elsewhere? What 
changes have there been in the quality of the workforce over time? Are 
those leaving government different in quality from those who do not 
leave? gao*s design proposes that new data on indicators of quality ^3 
gathered from samples of employees using surveys. This approach 
would for the first time provide regular, comparable information 
addressing each of the three questions for major occupaticxis in the pro- 
fessional and administrative workforce. The costs that can be estimated 
appear to be modest, less than $1 million for an initial cycle of data- 
gathering and analysis. (See appendix III.) 

The assessment of the newly-hired part of the workforce poses special 
challenges, since proper analysis requires a comparison group of non* 
federal employees in the same occupations, gag presents several options 
for methodologies of obtaining such a sample and the costs are uncertain 
since nothing similar has been done before. The data would help answer 
such questions as how people the federal government hired in specific 
occupations compared, in terms of educational achievement or experi* 
ence, to those entering similar jobs in non-federal sectors. (See chapter 
4.) 

Evaluation of change over time in the quality of the current workforce 
is the central element of the gao assessment design, involving surveying 
a sample of current employees at three levels of experience in 23 occu- 
pations. The data would answer such questions as whether, over time, 
federal workers in specific jobs show changes in the extent or quality of 
formal education they bring to their work, the extent or quality of con- 
tinuing education and training, whether over time those in an occupa- 
tional leries are generally more or less experienced, and to what extent 
the workers and their supervisors believe that individual workers' capa- 
bilities are adequate to their jrbs. (See chapter 5.) 
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The third question, about the quality of those separating from federal 
jobs, requires a comparison with employees of similar experience levels 
remaining in similar jobs. Data from this part of the assessment would 
answer questions about whether federal occupations were losing indi- 
viduals with higher qualifications, faster track records, and greater past 
recognition than those retained. (See chapter 6.) 

GAO suggests that a lead agency be selected on criteria of technical 
expertise, access to the data bases needed, i:;dependcnce, conunitment to 
the effort, and availability of resources. The core assessment covering a 
consistent set of occupations should be repeated at least every four 
years to provide trend data on changes in quality. (See chapter 7.) 

GAO concludes that methodologically adequate answers to questions 
about the quality of the federal workforce and changes in it over time 
will not come from existing p^ ^nnel records or existing productivity 
studies. Useful data could, however, be provided by a new program of 
data-gathering and analysis, established at congressional direction and 
with sufficient resources to carry out the work in a methodologically 
sound way. 



Reconunendation 



GAO reconunends that the (Congress authorize a continuing program to 
assess workforce quality based on the framework outlined in this report. 



Agency Comments 



GAO discussed the proposed assessment with officials at the Office of 
PersOiinel Management and the Merit Systems Protection Board. The 
officials in general raised no nuyor problems with the proposed defmi- 
tion of quality, the initial indicators, or the broad feasibility of the 
designs for the three segments of the assessment. However, in accord- 
ance with the requester's wishes, we did not obtain written comments 
on a draft of this report. 



ERLC 



Pages 



GAO/PEMMS-a? A Fraawwork Pgr Soidying the Pedml Workfom 



Contents 



Executive Summary 



Chapter 1 lo 

Introduction workforce Quality: Glimpses of the Information Problem ^ 11 

Objective, Scope, and Methodology 13 

Organization of the Report 15 



Chapter 2 le 

The Definition of Po? s^ble Denmtions 16 

Oi tv Assessing Quality in Individuals in the Workforce 22 

^JUailLy Definition of Workforce Quality 25 

SpeciHc Indicators of Quality 26 

Summary and Conclusions 29 



Chapter 3 31 

Using Administrative D^ta That Are Available in Agency Record Systems 31 

Wx\fx tn A ccMG FeasibUity of Enlarging Administrative r ^ 34 

tu /\2>^c^>^ D^^^ r^^^ ^ Available in 0PM Record Systems 36 

WOI Id'OrCe Quality Modifications in 0PM Data 37 

Summary and Conclusions 38 



Chapter 4 39 

Assessing the Bantering Questions About the Entering Workforce 39 

Workforce Needed to Answer the Questions and Potential 40 

Sources 

Proposed Design 43 

Summary and Conclusions 48 



Chapter 5 50 

Assessing Changes in Questions About the General Workforce 50 

the Workforce Needed to Answer the Questions and Potential 51 

Proposed Design 53 

Summary and Conclusions 57 



5 

O Page 6 GAO/PEMM8-27AFranefPOfkPorSCadyiiigtliePcd^ 

ERiC 



Contents 



Chapter 6 58 

Assessing Separations Questions About Those 'saving the Workforce 58 

From the Workforce Needed to Answer the Questions and Potential 59 

Sourc6s 

Proposed Design 62 

Sununary and Conclusions 64 



Chapter 7 eg 

Strategies for organizational Arrangements 67 

Tmnlpmpntinff thp Funding for the Assessment 70 

uiipiciiit^iitiiig UlC Tij^ Phasing of the Assessment 71 

Assessment conclusions 72 

Recommendation to the Congress 73 

Agency Comments 73 



Appendix I: Congressional Request Letter 74 

Appendix 11: Selection of Occupations for Sampling 76 

Appendix III: Sample Sizes and Costs 79 

Appendix IV: Advisory Panel Members, and Others 86 
Consulted During Our Work 



Tables Xable 2.1: Our Evaluation of Types of Measures for Three 28 

Dimensions of Workforce Quality 
Talile 2.2: Potential Indicators for Different Parts of the 29 

Definition of Workforce Quality 
Table 3.1: Number of Agencies Responding Positively to 32 

Questions About Workforce Data Elements 
Table 3.2: Number of Agencies Citing Barriers to 35 

Enlarging Automated Personnel Data Systems 
Table 3.3: Number of Agencies Citing Specific Concerns 36 

About Enlai'ged Data Systems for Monitoring Quality 
Table 4. 1 : The Feasibility and Interpretability of 41 

Alternative Data Sources for Assessing New 

Employees 

Table 4.2: Dimensions, Indicators, and Data Sources for 47 

Assessing New Employees 
Table 5. 1 : The Feasibility and Interpretability of 52 

Alternative Data Sources for Assessing the General 

Workforce 

Table 5.2: Dimensions, Indicators, and Data Sources for 55 
Assessing the General Workforce 



9^ 7 GAO/PEIfI>«.27 A FtaMfPoHc For StmAfIng tU Federal WorkTm 



Contents 



V Table 6.1: The Feasibility and Interpretability of 61 

Alternative Data Sources for Assessing Separated 

and Retained Employees 
Table 6.2: Dimensions, Indicators, and Data Sources for 64 

Assessing Separated and Retained Employees 
Table 7.1: Design for an Assessment of Federal Workforce 66 

Quality 

Table 7.2: Relative Difficulty of Parts of the Assessment 72 
Table U. 1 : Occupations Selected for Quality Assessment 77 
Table JI.2: Additional Occupations to Be Grouped for 78 

Quality Assessment 
Table III. 1 : Sample Sizes Needed for Three Segments of 81 

the Proposed Assessment of Federal Workforce 

Quality 

Table in.2: Estimated Costs of the Proposed Assessment 85 
of Federal Workforce Quality 

Figure Figure 2. 1 : Workforce Quality and Its Larger Context 17 



Abbreviations 

BLS Bureau of Labor Statistics 

CPDF Central Personnel Data File 

EPA Environmental Protection Agency 

uAO General Accoimting Office 

HHS Department of Health and Human Services 

MSPB Merit Systems Protection Board 

NASA National Aeronautics and Space Administration 

OPM Office of Personnel Management 

PACE Professional and Administrative Career Examination 

VA Veterans Administration 



GAO /PEMim-27 jmkinemKk For St»dy!iig tke Federal Worltfoffee 



Chapter 1 



Mtroduction 



There is a long history of concern about the quality of the federal 
workforce. The statutory merit principles require that "selection and 
advancement should be determined solely on the basis of relative abil- 
ity, knowledge, and skills." The federal government has invested great 
effort and has provided leadership to both the public and private sec- 
tors in developing methods for fairly assessing individual qualities to 
sApport selection of staff based on merit alone. Beyond selection, the 
present or desired quality of the workforce also is discussed in the con- 
text of many other personnel policy issues including setting pay. classi- 
fying positions, evaluating performance, linking pay with perfonoance, 
and def Jiing the benefits that will be offered those who are retiring. 
Workforce quality is brought into the more general debates about the 
ability of government to perform its missions, especially as government 
increasingly competes for employees in shortage occupaticms.' 

Despite the frequency of discussion of the subject, no statutory guide- 
lines exist that direct comprehensive or regular assessment of the qual- 
ity of the federal workforce. Rather, it is analyzed using individual 
anecdotes, unsystematic sampling of employee or supervisor opinicms, 
or by inferences from other data such aa comparisons of federal and 
nonfederal employees' pay and benefits. Such evidence has been used to 
make claims that the federal goveinmpnt is not attracting as strong can- 
didates for jobs as it used to, that the overall quality of the workforce is 
declining, or that highly valued employees are leaving government in 
recent years at a rate that should be cause for alarm. 

The Committee on Pest Office and Civil Service of the House of Repre- 
sentatives concluded that discussion of many topics in federal personnel 
policy could benefit if better information could be obtained on the qual- 
ity of the workforce, both its current status and changes over time. The 
committee request (see appendix I) asked us to examine the feasibility 
of setting up such a data base and, if it seemed feasible, to propose a 
design. 



^The most recent listing of GAG work on federal personnel management is in U^. General Accounting 
Office. Fecteral Personnel: Annual Iteiwrt on Acdvitics of OPM and MSPB, Fiscal Year 1986 , GAG/ 
GGM7-57 Cwashington, D.C.: May 1987). The CcMnptroUer General testified on GAO work related to 
revitalizing public service (GA0/T-GGD^21. March 24, 1988). 
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Chapter 1 

Introdaction 



Scattered efforts to evaluate the quality of the workforce, although 
intriguing and provocative, also suggest the problems of attempting the 
task. We found the following examples of assessments of quality in 
groups entering federal jobs, in the workforce n^ore generally, or in 
those leaving federal jobs: 

The Internal Revenue Service studied the quality of their incoming reve- 
nue agents, ccxnpared to new staff of accountmg firms, by giving a 
national test of accounting knowledge and found that 84 percent of the 
comparison group outside of government scored higher than the mean of 
the federal group in each of two years of testing. 

The Merit Systems Protection Board (mspb) examined the quality of the 
workforce by asking managers' views on whether the quality of appli- 
cants for professional and administrative jobs at entry-, mid-, and senior 
levels had changed in the four years preceding the 1986 survey, and 
found that 35-36 percent believed it had worsened at each level; 33-40 
percent believed it had stayed the same; and 24-31 percent believed it 
had improved (the percentages varied slightly across the levels included 
in the questicm). 

We attempted to profile the procurement workforce by using available 
automated i)ersonnel files and reported that there was information from 
a data base in the Office of Personnel Management (opm) on employees' 
education at the time of hiring that showed an increase over time in the 
proportion of employees with college degrees. However, no data were 
available on current status or trends in other indicators believed to be 
essential, such as experience and training. 

We reviewed the quality of technical staff assigned to the complex tasks 
of environmental cleanup in the Superfund section of the Environmental 
Protection Agency (epa) by asking employees to give opinions of 
shortages of critical skills and found that 39 percent of employees had a 
problem assuring the quality of their work because of a lack of skilled 
personnel. 

The Congressional Budget Office studied the quality of managers and 
sui)ervisors who left government compared to those who stayed by 
examining thdr performance appraisal scores and found that because 
those with lower ratings left at a higher rate than average, perhaps 
turnover was leading to separation of the less satisfactory performers. 



Workforce Quality: 
Glimpses of the 
Information Problem 
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The Department of Defense studied the quaJity of science and engineer- 
ing personnel who left the department, compared to those who 
remained, finding that separation rates of those who had scored above 
650 on the mathematics part of the Scholastic Aptitude Test (at the time 
of graduating from high school) were about 50 percent higher than 
those of persons who scored below that level. 

We assessed the effectiveness of a wide range of human resource man- 
agement policies by surveying a government-wide sample of personnel 
officers, including one question about their opinions of changes in their 
ability to acquire and retain quality persor in the last five years, and 
found that half believed their ability to acquire good people was worse 
or much worse and over 60 percent believed their ability to retain such 
people was worse or much worse. 

We studied whether exceptional Marine Ciorps officers were being 
retained in the Corps by examining performance appraisals and reenlist- 
ment records and found that the service did not lose its best men, at 
lea^t not during the period reviewed. 

Although useful for other purposes no doubt, the studies from which we 
drew these examples higlilight a number of difficulties in puUing 
together current information forjudging the quality of the wc'kforce. In 
addition to asking different questions, the studies show great divei^ity 
of approach. They have not been guided by any l?vy or regulatioii, have 
not been performed at predetermined intervals to allow trend analysis, 
have no common unit of analysis (having examined single occupations 
or clusters, at different levels of aggregation within part of one agency, 
within a department, or across government), have had different compar- 
ison groups or none, and have not been consistent in the indicators 
included in the defmition of quality and how they were measured. Fur- 
ther, questions can be raised about individual indicators used in the 
studies. For example, is very much learned about the quality of expe- 
rienced employees by examining their college entrance test scores? Do 
genera, .asts of accounting knowledge, even when developed by the 
major national professional organization, tell us much about quality in 
relation to the tasks of a federal revenue agent? Are managerial per- 
formance* appraisals done in different ways in different agencies com- 
parable? Do we gain useful information from managers' general opinions 
about such a broad question as the quality of applicants or new hires, 
without reference to any occupation or to any standard of needed qual- 
ity? Though these studies may have served other purposes well, they 
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demonstrate the problem of trying to understand workforce quality 
from the existing fragmented sets of data. 

Our challenge was to imagine an alternative information base that 
would remedy the many shortcomings of measurement, design, and 
interpretation that could be pointed out in the information now 
available. 



Objective Scope and objective of our effort, then, was to develop a means of routinely 

li/r ^ l' providing Congress and other parts of government with aggregated and 

MetnOuOlOgy analyzed descriptive data on the quality of the workforce over time. 

The committee chairman's request referred to the civilian workforce in 
general terms. After agreement with committee staff, we confmed our 
work to considering assessment of the executive branch civilian non- 
postal workforce. Thus we did not consider designs for assessing the 
quality of presidential appointees, the Senior Executive Service, the uni- 
formed military, or legislative or judicial branch employees. We also 
agreed to focus only on professional and administrative occupational 
series within the civilian executive branch workforce. (The other two 
types of occupational series are technical and clerical.) 

To carry out this objective we developed a definition of quality based on 
discussions with experts in measurement, federal employee groups, and 
with experienced personnel managers from federal and nonfederal sec- 
tors. (See appendix IV for a list oi persons and organizations consulted.) 
We then surveyed eight federal agencies to discover the extent to which 
• their civilian personnel records contained data that matched our defini- 

tion of quality .2 We asked the agencies also about any other a<*va they 
had pertinent to alternative definitions of quality that we should con- 
sider. If few data were available, or were inaccessible, we asked about 
the feasibility of improvements in the agencies' oversdl data on their 
employees. We reviewed the data maintained in the opm Central Person- 
nel Data File (cpdf) and the feasibility of additions to that data base. 



^The eight departments or agencies were: Air Force, Environmental Protection Agency, Health and 
Human Services (HHS), Interior, Justire, National Aeronautics and Space Administration (NASA), 
Navy, and Veterans Administration (VA). We chose agencies on two criteria: first, that they included 
substantial numbers of employees in a range of professional and administrative jobs, and second, that 
(in the cases of EPA, NASA and VA) they included occupations of special interest such as scientists, 
engineers, and health specialties. ^ ^ 
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To go beyond information now in pgency records, we developed an alter- 
native approach based on gathering new data from samples of federal 
employees and, where needed, comparison groups of other employees. 
To develop the proposal, we reviewed the public discussions of 
workforce quality and determined the main distinctive questions com- 
monly addressed so that our design would allow the production of 
responsive information. For each question, we reviewed technical mea- 
surement alternatives including individual tests and measures used by 
opm; existing surveys that could, with some augmentation, serve our 
purpose; opm statistical data on the segments of the workforce needing 
to be assessed for each question; prior efforts to measure the various 
elements of our definition; statistical sampling requirements; and the 
costs and feasibility of our approach to each of the questions. 

We selected an advisory group to review our work. The group included 
former high-level federal officials; federal agency personnel executives; 
an employee group official; experts in the disciplines of political science, 
evaluation, labor economics, and psychological measurement; a former 
member of Congress who was also formerly ^ <^^ate human resource 
executive; and an executive and personnel ai. . ^ssment expert from the 
private sector. We obtained advisory conunents f^^om these ^consultants 
on our overall approach to the project at the outset, and we reviewed 
the proposed assessment questions and designs with them as we com- 
pleted our analysis. They also reviewed the draft report. Appendix IV 
lists the members of the group. 

Our approach has several strengths, including the detail of our review of 
existing data stored in selected agencies' computer files and paper 
records and the breadth of our discussions with interested groups and 
individuals about issues in defining or measuring quality. One limitation 
is that we cannot suggest a design representing a rigorous consensus of 
experts' views, as we did not do a general national survey of opinions on 
the best way to approach the taslc. Another is that our design work 
addressed only the full-time permanent professional and administrative 
segment of the overall federal executive branch workforce. Although we 
believe it is one of the most important parts, and the appropriate group 
on which quality assessment should begin, it could be useful to devise 
relevant measurements and research designs to track the quality of 
those in appointed positions, in the Senior Executive Service, in clerical, 
technical, and wage-grade jobs, or with temporary or part-time 
appointments. 
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Chapttfl 
Introduction 



Organization of the 
Report 



Chapter 2 reviews a range of possible areas of measurement that could 
be included in a defmition of quality, gives the definition we propose, 
evaluates general types of data that might be used in measuring the ele- 
ments of the definition, and concludes with specific indicators we looked 
for in agency data. Chapter 3 reports our review of personnel data 
maintained in agency records and our evaluation of its usefulness for 
measuring the proposed definition. Chs^ters 4-6 present three distinct 
questions about the workforce and designs that should provide useful 
data on each. Chapter 7 suggests some issues of implementation of the 
assessment. Appendices include: the congressional request letter (appen- 
dix I), occupations proposed to be sampled (appendix II), sample sizes 
and estimated costs for an initial cycle of the assessment (appendix III), 
and those we consulted (appendix IV). 
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Chapter 2 



The Definition of Quality 



The examples of scattered quality measurement efforts cited in chapter 
1 show the variety of defmitions and measures of quality that can be 
used, including individuals' knowledge (as demonstrated on a test) or 
job performance (as assessed by supervisors), or process measures such 
as difficulty of recruitment or retention (measured by managers' opin- 
ions). We found widespread belief that any assessment should start with 
a multi-facetecl definition and that individual capacities should not be 
assessed in isolation from specific work tasks. 

This chapter is organized into three sections. The next section examines 
the concept of a quality workforce and the wide range of information 
that has been used to address related questions and highlights the focus 
on individuals that guided the definition of quality proposed for this 
assessment. The second section analyzes various dimensions that could 
be included in the definition and gives the proposed two-part definition 
of a quality workforce as the possession of strong capabilities (general 
and job-specific) and the degree to which those capabilities are consis- 
tent with the needs of the specific job. The last section evaluates types 
of data that could be used to measure aspects of the definition and pre- 
sents a list of specific data elements for which we searched in existing 
personnel data. 



It is at first tempting to try to include in the definition of quality the 
wide range of continuing concerns about government: for example, its 
overall efficiency and productivity, the integrity of its officials, or the 
innovativeness of its scientific and engineering endeavor. The more the 
definition of quality is broadened, however, the more difficulty there is 
in measurem.ent and interpretation. Restricting the assessment to indi- 
vidual characteristics provides a complex challenge in itself. Neverthe- 
less, we began by considering a wide range of ideas that could be part of 
the definition of the quality of the federal workforce. 



Possible Definitions 



The federal workforce at any point in time is both a consequence of sev- 
eral environmental influences and a cause, along with other factors, of 
the results of government. We developed a model of the workforce, 
shown in figure 2. 1 , and examined each of the five elements of it for 
relevance to our assignment and for feasibility of measurement and 
interpretation. The shaded section of the figure, labelled workforce 
quality, including attributes of workers as individuals, became the focus 
of our measurement effort. The four sections that follow describe 
aspects of quality found in the four areas of the model that we eventu- 
ally eliminated from consideration for inclusion in the definition: the 
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^Proposed tndicators of workforce quality discussed later in this chapter are drawn frorr. the area high- 
lighted by snading 

external context, the i)ersonnel management context, the immediate 
work context, and the outcomes of work. 



The External Context 



Several aspects of the broad environment in which the federal govern- 
ment competes with other employers for v/orkers have been suggested 
as areas for improved measurement in order to understand workforce 
quality. For example, there are i)eriodic shifts in public views about the 
role of government and about work and careers. As government 
responds to these chaitging ex!>ectations, there will be changes in the 
kinds of jobs that are available. In addition, there are shifts in the kinds 
of rewards people want from jobs. Both will influence job-seekers' 
views of the attractiveness of government work, and, hence, the 
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nature of the applicant pool. Constantly rising costs of education are 
one specific influence affecting graduates' choices of jobs in recent 
years.* 

The opinions of students, higher education faculty, or the public about 
different employers and different kinds of work can be readily studied. 
However, the importance of these views, among other influences on 
actual job choices, is unknown, and the data offer no direct answer to 
questions about workforce quality. Once particular quality problems 
have been identifled, such as in particular occupations or career stages, 
then targeted studies of the external context can be designed to search 
for possible causes, if desired. 



The Personnel Common substitute measures for workforce quality are the processes of 

Management Context workforce planning, and the steps of recruiting, selecting, and classify- 

ing workers. In this view, one has only to judge the degree of indepen- 
dence, fairness, and meritocracy built into personnel management to 
have a yardstick for the quality of the resulting workforce. 

It would be possible to evaluate the planning stage, examining the preci- 
sion and validity of agencies* deflnitions of staff needs and the extent to 
which any deflnition of quality was used in workforce planning. In later 
stages of personnel management, managers and personnel staff adver- 
tise, recruit, interview, examine, and select individuals to meet those 
needs. Evaluation again could examine procedural details of each of 
these steps that affect the workforce actually hired and set to work. 
Feasibility, then, is not the problem here. Rather the dif Acuity is that we 
cannot be sure of the degree to which the quality of personnel manage- 
ment does in fact determine the quality of the workforce. That is, as 
with data on the external context, such information can only answer 
questions about workforce quality indirectly. Yet the need here is for 



^The external context that affects workforce quality can be conceived even more bfxradly to include 
the biological world we inhabit. One search for explanations for a decline in the quality of military 
recruits related that to a wide range of other health and environmental degradations ranging from 
maternal and preniatal nutrition and changes in methods of delivering babies, to atomic fallout, diet, 
and food additives. The authors of this analysis concluded that this ecological context was the proper 
level of analysis and that **the quality deciine...i8 an expectable and perhaps inevitable consequence 
of an accumulation of ecological changes that have occurred throughout the United States in the past 
several decades." The level of context at ifihkh one seeks causal factors clearly affects the search for 
remediet as well; these authors argued that the only proper response to the qiwlity problem was to 
focus on biologkra] research and interventions centered on diet and nutritkm. (Bernard Rimland and 
GeraM Larson, *The Manpower Quality Decline: An Ecoktgkal Perspective/* Armed Foroes and Sod- 
e^.8(FaUl98l),2l7a) 
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more direct indicators about the workforce, to facilitate agreement 
about results. 

Independent sources of indicators are needed for a second reason, to 
provide an independent way to evaluate the processes themselves and 
especially variations in their quality. The most convincing causal evalu- 
ations would link measurable differences in the steps of personnel man- 
agement to differences in work results (for example, to show that better 
selection tests result in a workforce better matched to tasks, which in 
turn is more productive than one less carefuUy selected). But in the 
absence of such strong study designs and the diverse data they would 
require, at least one would like to know, for example, whether increased 
effort in recruitment or more careful screening was associated with any 
noticeable change — that is, was there any contemporaneous change in 
any index of workforce knowledge, skill, or any other quality indica- 
tors? Thus it seemed important to develop an assessment design that has 
measures independent of personnel management, first, to provide a 
long-term set of indicators that can reveal broad problems to which 
management could respond, and second, to determine whether quality 
indicators show any trends that can be associated with government- 
wide management changes. 



The Immediate Work As in any workforce, federal employees are given specific work to do; 

Context ^ supervised, offered short- and long-term rewards for 

their efforts, given resources to attain governmental objectives, and pro- 
vided some degree of stability and support in their work environment. 
Each of these dimensions is variable and each can serve as an explana- 
tion for the quality of workers attracted and retained and for the qual- 
ity of their work output. 

For example, employee groups emphasized in interviews the importance 
of management in designing work and creating the conditions for effec- 
tive or ineffective use of a workforce of whatever skill level. (Classifica- 
tion of jobs is said to be difficult to change quickly enough to keep pace 
with changing work.) Federal managers emphasize the fluctuating 
resources they are given and the shifts in agency policy that affect their 
efforts to deliver. More generally, there is wide discussion of the effects 
of newly-revised federal retirement ber fits on who will be attracted 
and retained, of federal pay and its comparability to other sectors, and 
of other rewards such as promotions and merit increases. Tasks and 
results can be affected by the degree of thoughtful management evalua- 
tion to correct course based on past experience. All of these aspects of 

20 
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work setting and management may influence the quality of employees 
attracted and retained by the federal government and their work per- 
formance. Such analyses are not inappropriate; indeed, they are crucial 
in understanding how the government makes use of all of its resources, 
including its employees. But again, data on all these details of the con- 
text of management and leadership do not provide indicators pertinent 
to the committee's specific interest in assessing the workforce. 



Outcomes/Productivity The workforce is set to the wide range of government tasks, for example 

collecting taxes, writing checks, exploring space, gathering statistics, 
delivering mail, forecasting the weather, and keeping airplanes sepa- 
rated in the sky. It is appropriate to assess the quality of performance 
or results in each case.^ We have called repeatedly for measurement of 
outcomes as an important source of data to improve government man- 
agement, and ai^ably there could be no more important measure of the 
quality of the federal workforce than its performance.^ 

We decided not to propose measures of performance or outcomes in our 
response to the committee's request for two reasons. First, such data are 
difficult to gather even in one occupational area, and the assessment 
design needed to cover a spectrum of professional and administrative 
work. Second, outcome data could not be readily interpreted to show the 
quality of the human resources available for the government's tasks, 
which is the committee's interest. That is, it would be incorrect to draw 
conclusions about the workforce from data on work results, since those 
are affected by many factors in the situation (as outlined in figure 2. 1) 
in addition to the knowledge, skill, ability, and other characteristics of 
the workforce. 

Also, as in the case of personnel processes, accurate and comprehensive 
information on the workforce is important to collect in order to use it in 



2As shown in figiire 2 1, jtome intermediate outcomes include whether employees show up regularty, 
in condition to perform the assigned work, and stay with the job over a period sufficient to recoup the 
costs of getting them on board. As they attempt to produce results, employees spend time on or off 
task once they get on the job. And when pe<^ get to work, they produce results by th^ 
groups, and in laiger units. These intermediate outcomes ooukl all be evahiated through objective 
data. However, most people probably have direct senices in mind as outcomes, or more generally, 
attamment of government missions. 

''See, for example: VS. General Accounting Office, Ways to Improve Federal Management and Use of 
Productiv^Baa^ Reward Systems, GA0/FPCIVfll-S4 (Washington. 5!C.: Dec 3l . iflflO) and 
IncrMsed f^Py^y^ty Jjti^^ Can Help Control Government Costs , GA0/AFMM4-1 1 
(Washington, D.C.: Nov. 11, 1963). iMrect measures of the perfonnance of soniegovenunentfiinctions 
are poasibte, as shown in productivity data reported by the Bureau of Labor Statistics. 

2x 
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analyzing variations in productivity data gathered separately. When 
problems with the outcomes of government are noted, where should we 
look to find what should be done? It would be useful to know whether 
changes had occurred in workforce quality, so that one could assess 
staffing as an area for added effort if outcome pleasures showed needs 
for improvement That is, if there are smaller returns in tax audits, too 
many mid-air near misses, or increased patient complaints about hospi- 
tal nursing care in government facilities, can these be understood better 
with reference to changes in the characteristics of the revenue agents, 
controllers, or nurses, or with reference to other factors (such as their 
assignment patterns and overwork, quality of supervision, effectiveness 
of technology, and so on)? 

We are proposing, in the definition outlined in the rest of this chapter 
and in the specific quality assessment efforts discussed in chapters 4-6, 
to focus on numerous attributes of the workforce. It is important to rec- 
ognize, however, that their precise contributions to performance and 
outcomes are not settled by empirical research results, though they are 
plausible and widely discussed as important courributors. That is, it is 
stUl only an assumption — although a higiily rea^ onable one — that 
efforts to increase the level of those attributes (s\ch as by increasing 
pay or benefits in hopes of attracting or retaining workers with desired 
characteristics) will lead to improvements in work performance. Since 
there does exist some uncertainty over the importance of human 
resource variables for performance, it would be very useful to link the 
workforce quality assessment we are proposing to a continuing broader 
program of evaluation of performance, as well as research on its various 
causes. 

We decided, however, to design the workforce quality assessment sepa- 
rate from such an overarching program of research on productivity 
because of cost, feasibility, and measurement uncertainties. Some people 
did recommend to us that we try to link descriptive data on the building 
blocks shown in figure 2.1 on individuals and their contexts together 
with measures of work performance and outcomes, to produce a causal 
analysis that could explain the contribution of each to the outcomes 
observed. Such a process of analysis is of course done in less formal, 
nonquantitative ways by a wide variety of actors and observers who 
weigh many factors to reach conclusions about how to use resources to 
accomplish government goals. But tid}ring all the many factors into an 
empirical model seemed extremely difficult, given the current scate-of- 
the-art in measurement for most parts of the model, and non-quantita- 
tive models are not easy to use for recognizing changes over time. This is 
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not to say that such an analysis should not be planned for the future, 
given continued improvement in measures of both workforce attributes 
and other inputs as well as improved outcome measures. For the pre- 
sent, however, we conclude that such an effort is probably not feasible 
with current levels of scientific measurement and analysis/ 



The Individual as the 
Focus of Workforce 
Quality 



In sum, although there are important questions in each of these four 
areas of context and outcomes, the most appropriate focus for initial 
measiuement still appears to be the individual employee. Even though 
the characteristics of the individual worker form only one building block 
among the many needed for complete understanding, no other domain 
appears promising to measure in response to the questions of likely 
interest to the committee. Thus we concluded that our design for assess- 
ment should center on an initial set of employee variables alone. If that 
assessment proves workable and useful, it will both stimulate, and form 
a key starting point for, wider analyses of the causes and consequences 
of workforce quality. 



AssPSSintf Oll5i1itv in ^ enormous variety of individual characteristics of workers could be 

T^r^^ 1 • u assessed. Considering the specific focus on the workforce as individuals, 

IncUVlaUcilS in the we examined various frameworks to develop a detailed definition. 

Workforce 



Diverse Concepts of in the broader literature many conceptualizations of human qualities 

Individual Quality ^^ist, including those of: 

• the economist, who makes comparative evaluations of the ''human capi- 
tal" in different labor forces (such as in different parts of the world) 
with relatively crude numbers representing schooling and work 
exi)erience; 

• the psychologist, trained to judge individual mental ability, personality, 
motivation, or overall mental health with written, face-to-face, and situ- 
ational measures; 



^The field of personnel psychology has contributed many studies that are steps in the same direction, 
examining the validity of various measures of people's characteristics in predicting later events at 
work. The daU against which the predictions are tested, however, are often only uidirect measures 
such as tenure and turnover (how long a worker sUyed on thejob) or opinions (supervisors' ratings) 
rather than work results. Advances beyond such work require more concrete measures of work 
results and measures of a more compl^ set of causal factors (in addition to empkyyee attributes) to 
explain the resiflts. 
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• the business analyst, trying the new specialization of human resource 
accounting and attempting to value a firm's human assets for a balance 
sheet; or 

• the journalist, drawing on interviews, observation, history, philosophy, 
and current opinion polls in writing for the general public about the 
character of a public servant. 

In the specific federal context, the first of the statutory merit principles 
included in the Civil Service Reform Act of 1978 states that "selection 
and advancement should be determined solely on the basis of relative 
ability, knowledge, and skills," which suggests some descriptors of qual- 
ity that might be measured. 

The two msgor criteria we used in examining potential constructs and 
measures for our deHnition were 

• the feasibility of measurement in the federal setting (whether such data 
could be readily obtained on sizable numbers of people at dispersed loca- 
tions) and 

• the interpretability of the data (whether relatively clear inferences 
could be drawn from data on an indicator). 

FYirther, we examined whether our defmition should be influenced by 
the strict legal tests that apply to definitions and indicators of quality 
used in individual hiring and promotion decisions. 



The Legal Standards of The legal standards i)ertinent here reflect, on the one hand, continuation 
Definitions original aim of all merit selection methods by demanding closer 

scrutiny of whether tests or other screens fairly assess individuals ' fit- 
ness for a particular job. The trend here is to require ever-stronger dem- 
onstrations that a specific quality dimension (for example, verbal 
ability), and the specific measurement of that dimension (in the same 
example, a pai)er-and-i)encil reading comprehension or verbal analogies 
test), are related to performance of the tasks of a specific job. 

On the other hand, other laws and court cases provide for review of any 
quality measurement if its effect, when used for selection, seems unfair 
to groups in the society, such as women or racial minorities. Thus, qual- 
ity dimensions and measures chosen by employers with an rye to effi- 
ciently matching jobs and individuals face review on equity grounds as 
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well as for evidence that they validly reflect the work.*^ The federal gov- 
ernment discontinued the use of a test called the Professional and 
Administrative Career Examination (pace) because of concern that 
minority groups scored disproportionately poorly on it. 

The monitoring effort we were asked to design does not face the same 
rocky road as tests or other data used for employment selection. 
Because the information would have no impact on individuals, the valid- 
ity of the assessment results would be tested not so much by the compet- 
ing and conflicting legal standards of fairness just described, but by 
tests of practicality and usability by the consumers, who are policy 
makers in the legislative branch, leaders and managers of the workforce 
in the executive branch, and observers of government generally. 

An example of the distinction between using indicators about individu- 
als to predict something about them and using such data to monitor a 
larger system comes from the field of education. Concern about the dra- 
matic downward slide in scores on the Scholastic Aptitude Test taken by 
high school students in the 1970s did not arise from, nor was it settled 
by, technical arguments over the test's precision in predicting college 
success (though that issue was of course raised). The aggregate figures 
were taken as a kind of national thermometer that was indicating some- 
thing important about the health of education. Viewed that way, the 
data stimulated much further analysis of the nature of school achieve- 
ment, its proper measure, and its multiple causes, and served as part of 
the impetus for a school reform movement that is still under way. In 
sinular fashion, indicators of the quality of the workforce, even though 
not suitable as predictors about individuals, may nonetheless orovide 
useful signals about an aggregate of employees and should provoke 
more detailed study to understand the message of initial warning flags. 

Proper analysis and inference is a serious issue with respect to any pro- 
posed set of information. The issue of assigning the assessment work to 
an organization capable of credible interpretation of the assessment 
data is raised in chapter 7 in the discussion of implementation topics. 



'^The Committee on Ability Testing of the NationaJ Research Counal analyzed the dilemmas of testing 
for abilities in education and employment in light of these two different legal standards. See Alexan- 
dra K. Wigdor and Wendell R. Gamer (eds ), Ability Testing (Washington. DC: National Academy 
Press. 1982). 
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Definition of 
Workforce Quality 



We ended our analysis of focus, feasibility, and measurement validity by 
centering the assessment on two broad domains of quality. We define a 
quality professional and administrative workforce as one with: 

strong capabilities (both generic and job-specific) and 
that degree of capability needed on a particular job. 

These two complementary aspects of the definition of quality, using 
very different standards as their bases, emerged from our consultations 
with those who would either contribute data to the assessment or use 
the results. On one hand, quality can be defined as the extent ti which 
an individual has desired characteristics such as education, job related 
experience, specific skills, motivation, or even certain personality traits. 
Gathering such data and arraying results by groups (such as how much 
education workers in a particular job have, or workers in a particular 
agency, or workers at a similar level of experience such as newcomers), 
carries the unstated assumption of a generic standard, that more of any 
characteristic would be desirable in the group. This approach to defining 
quality is exemplified by the question "Is the federal government 
attracting the best and the brightest?" 

However, from a strategic point of view it is of equal or even greater 
interest to assess quality against a standard of need. That is, a manager 
thinking of particular tasks faces decisions about the amount of time 
and effort that should be invested in recruiting and screening, or possi- 
ble chariges in the conditions of work to consider in order to attract, 
workers with the desired degree of education, experience, and other 
qualities. Managers we snoke to often noted that they had a need stand- 
ard, saying the equivalent of "We don't really have to attract the possi- 
bly idiosjoicratic superstar; to do ouf work well, of course we need a 
solid performer, but here's what we mean." In some other situations, 
success in extremely challenging work at some frontier may be attaina- 
ble only with the world's foremost experts. The general point is that the 
requirements of the job, within the context of the goals and plans of the 
organization, define a second standard against which a person's charac- 
teristics can be judged. This viewpoint seems important for both under- 
standing and action. Knowledge of individuals' capacities is interesting, 
but is limited in interpretability without this contextual reference point. 
An effective policy action would seem to be more likely when directed 
towards a specific target of obtaining the needed quality, rather than 
towards a vague goal of higher and higher levels of quality. 
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Although we began our definition by excluding the context in which the 
employee served, we concluded that this was unwise. First, the question 
of needed quality provides information about reality that corrects and 
completes the information about generic capabilities. Second, we think 
the basic concept of needed quality is important to include so tha"^ 
experience with its measurement can accumulate and blrategic judg- 
ments about needed human resources can be refined. In light of the 
uncertainty noted earlier in the chapter, about the precise relationship 
between individual attributes and eventual government performance, 
the proposed indicators of both absolute and needed quality should be 
considered provisional, subject to revision as empirical evidence shows 
particular indicators to be especially critica' 



Specific Indicators of 
Quality 



To begin to narrow our examination of alternative ways to collect data 
on the two-part definition we propose to use, we divided individual 
capacities into two areas: knowledge, skills, and abilities, and attitudes, 
values, and motivation. For these two, and the second part of the defini- 
tion (the match of indi\ iduds* capacities with the job), we listed a 
number of possible types of data. 



Evaluation of Types of Each type of data can be evaluated on two dimensions, the clarity of 

Data interpretation about the quality of the workforce that could come from 

such data and the feasibility of obtaining the data. Measures that 
seemed likely to result in ambiguous interpretations or that would be 
very difficult to obtain on the geographically dispersed federal 
employee group are not proposed for further active consideration at this 
point; the others are candidates for further exploration. (The analysis 
here is general; the specific relevance of parts of the data to different 
analyses of the workforce is discussed in chapters 4-6 below. For exam- 
ple, education indicators are most relevant for those new to the 
workforce and without work experience.) 

We concluded that direct tests of knowledge, skill, and ability (for exam- 
ple, using paper and pencil examinations or obs?rver i-atings of individu- 
als in simulated task situations) are not feasible, despite the strong data 
they provide, because they require controlled testing conditions that are 
impractical on large dispersed samples. A wide variety of less direct 
indicators, such as various details of education an x work experience, are 
conunonly discussed, however, as signals of quality. For example, a 
workforce may be considered to have gi eater knowledge and skill with 
members who have: ^ , 

^ 4 
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• more foiroal education; 

• better educational rcGi lts as shown by higher grades, test scores, or 
class rank; 

• more professional training, certification, licenses; or 

• more general or specialized work experience. 

We retained this category of data for further exploration. 

In the area of attitudes, values, and motivation, there are many possible 
topics on which employees could report their views (such as, for exam- 
ple, morale, commitment, or job satisfaction). However, interpreting 
them is less clear than with other kinds of indicators. Because these top- 
ics are widely discussed, however, we decided to retain the attitude type 
of data for further consideration, but chiefly to look for existing data 
sources (at agency level or government-wide) that would be adequate. If 
there were few such data, we would have doubts about investing much 
in new data-gathering of this kind. 

Attitudes and motivation towards work may reflect under ly'.ig person- 
ality dimensions and there are many measurement instruments and 
methods in that area. There is a historical body of literature discussing 
"types" of individuals attracted to, or produced by, long-term work in 
bureaucratic settings in government and elsewhere (for example, per- 
sons more interested in job security than in taking risks or in msyor 
achievements). Evaluating personality characteristics is, however, a 
complex measurement problem and interpretation of the results is con- 
troversial. We did not consider that type of measurement further for the 
initial assessment design. 

To assess the match of workforce quality to needed quality, it could be 
desirable to observe work performed by employees with varied charac- 
teristics, to empirically establish the capabilities needed (for acceptable 
output, for high output, etc.). This task could be part of agency person- 
nel management, but indicators requiring direct observation are not fea- 
sible as part of the overall assessment we are considering, in light of the 
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sample sizes needed and the dispeised workforce to be assessed." We 
propose that the initial design begin with data from employee self- 
reports, supplemented with a second viewpoint obtained from each 
employee's supervisor. 

Thus, after our evaluation, we retained three types of measures for fur- 
ther exploration to implement the defmition of quality, including (1) 
indirect measures of knowledge, skill, and ability; (2) general attitude 
measures; and (3) assessments by employees and supervisors of the 
match of personal capacities and the job. (Table 2.1 summarizes our 
evaluation of various measures and our initial conclusions about each 
type.) 



TaMt 2,1: Our Eviuation of TVpe» of Meamret for Thrte DImanrtons of Work! >rce Quality 



Df imnaion and measuras 


Clarity of 
intarpratation 


Faasibilityof 
maaauramant 


Initial coriclusion 


Knowledge, skills, and abilities 
Direct tests 
Indirect measures 


High 
Medium 


Low 
High 


Do not attempt 
Explore further 


Attitudes, values, and motivation 
Surveys of employee views 

Measurement of basic personality 
dimensions 


Medium 
Low 


High 
Low 


Avoid major investment, use existing data if 

possible 

Do not attempt 


Match of individuarcapacil.es and job needs 
Direct observations 
Self-assessment 
Supervisory assessment 


High 

Medium 

Medium 


Low 
High 
High 


Do not attempt 
Explore further 
Explore further 





The three types of potentially useful data shown in table 2.1 can take 
many specific forms. To check which of these are available in various 
sources, we developed a comprehensive list of indirect indicators for the 
knowledge, skill, and ability area, and general questions and probes in 
the other two areas. We used these as the core items we looked for in our 
review of agency personnel records, described in the next chapter. (See 
table 2.2.) 



^The difficulties of establishing workforce needs are sizable. G AO reviews of military manpower 
planning found the services have difficulties in establishing the basic nun^bers of workers needed, 
assuming uniform capability. Unsur orted Judgments were a chief source of the estimates, although 
GAO believed hard data were feasiuie to gather on many of the jobs reviewed. It seems likely to be 
even more difficult to empirically link different degrees of employee capabilit>' to differences in out- 
put. See U.S. General Accounting Office, Mafine 9?IP? y^"P9y?i jf"P'y^^'"^"j^ j^^!^^ 
for Dete> .nining Manpower Reouirements . GAO/NSlAD-87'102 fwashington, D.C.: May 1987) and 
Air Force Manpower Program: iinprovqnents Needed in Procedures and Controls , GAO/ 
NSIAM7-137 (washln^n, b.C: june 1987). 
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Tabto 22: Potential Indicators for 
DIffarant Parts of the Definition of 
Worliforca Quality 



Dimension 



Knowledge, skills, and 
abilities 



Indicators 



Education 



Years of schooling 



Degree(s) awarded 



Date of degree(s) 



Grade-point average 



Rank in class 



Inttitution attended 



Major f:eld of study 



College entrance test scores 



Other test scores 



Continuing education, training 



Quantity (hours, days, units, credits) 



Kind (course names) 



Source 



Professional certificates, licenses 



Examination scores (CPA, bar) 



Certification, licensure record 



Work expenence 



Goneral work expenence 



Specialized work expenence 



Promotion history 



Awards (monetary and other) 



Disciplinary actions 



Performance appraisal score 



Other managenal ratings 



Any other measure or indicator, either generally maintained 
on record or maintained for specific occupations 

(Open-ended probe for any existing agency survey or other 
data in this area) 



Attitudes, values, and 
motivation 

Match of individual capacities (Open ended probe for any data used by agencies in this 
and job needs area) 



Summary and 
Conclusions 



Because the definitions and concepts used by different observers in 
judging the federal workforce are so numerous and complex, some pri- 
orities must be set as to the most useful defmition for a beginning effort 
at assessment. As experience with an initial data set accumulates, addi- 
tional measures and further questions can be attempted. This chapter 
has outlined the following points. 
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First, of the many possible definitions of quality, a beginning assessment 
should focus on the individual and his or her work, excluding (leaving 
for later study) personnel processes, management and the work context, 
work outcomes, and the broader environment in which federal person- 
nel management operates. Assessment of performance and outcomes is 
of the greatest importance for improving government, but is a serious 
measurement challenge and does not respond directly to the committee's 
request as it would not provide clear indicators concerning the 
workforce. Data on individual attributes of those in the workforce need 
to be interpreted cautiously, because of uncertainties in the relationship 
between these qualities and work results. 

Second, once having deHned quality in terms of attributes of individuals 
in the workforce, the definition should include two ba«ic elements: that 
is, both strong capabilities and the degree of capability needed for the 
work. This definition emphasizes that individual characteristics should 
not be judged in isolation and underscores the long-term goal of linking 
quality assessment to strategic personnel decisions about the needed 
workforce. 

Third, there are types of data that are both generally feasible to obtain 
and relatively straightforward to interpret that measure aspects of the 
proposed definition. These include a variety of indicators of knowledge, 
skill, and ability, as well as measures of attitudes. The match of individ- 
uals to their jobs can be assessed through the views of both employees 
and sui)ervisors. In time, more direct measures in each area can be 
explored, along with exploring their causal links to work outcomes. 
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Using Aditiinistrative Data to Assess 
Workforce Quality 



Because federal agencies develop a large amount of information on the 
workforce in the course of routine personnel processes and agency man- 
agement more generally, we evaluated whether such administrative 
data could be the source of the information needed for assessing 
workforce quality. The principal criteria applied in reviewing 8 agen- 
cies' records and the central records kept at opm were comprehensive- 
ness of the quality indicators available and the accessibility and recency 
of da^^ on them. In addition to providing details of existing data, agen- 
cies estimated costs and other factors that would be involved in any 
attempt to change the existing data system to make workforce data 
more comprehensive, accessible, or up-to-date. 

This chapter Ls organized in two m^jor parts, the first summarizing case 
studies of data at 8 agencies, the second reviewing opm data. Each part 
presents the data now available as well as the potential for improving 
what is available. 



As described in chapter 2, we are proposing that a multi-faceted defini- 
tion of quality be used, in which three different types of information 
could be useful. These include indicators of knowledge, skill, and ability; 
information on attitudes, values, and motivation; and information on the 
match of individuals* capacities to the needs of a specific job. Eight 
agencies provided details of records they had on any of these types of 
measures.* 



Data on Knowledge, Skill, Table 3.1 shows the availability of 23 items of workforce information 
and Ability Indicators could serve as indicators of one segment of our definition of quality 

— individuals' knowledge, skill, and ability. About half, 13 of the 23 
elements, are stored in the records of 7 or 8 agencies, but adding the 
criterion of accessibility reduces the total. Only 4 data elements of the 
list could be easily retrieved from the records of 7 or 8 agencies, and 
these have significant limitatic ns. One common element is the duration 
of any training. This item (and the others on training) would be difficult 
to use because there is less central direction about record-keeping and 
reporting in this area with the result that agencies define training differ- 
ently and sometimes store the data in separate data systems apart from 
the main personnel records. (As discussed in the section on opm data 



*The eight departments or agencies were: Air Force, Environmeatal I^rotection Agency, Health and 
Human Serviceai, Interior, Justice, National Aeronautics and Space Administration, Navy, and Veter- 
ans Administration. 



Data That Are 
Available in Agency 
Record Systems 
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Tabto 3.1: N«iflPb«r of Agoncitt 
RMpondiiig PMlttvoly to Questions 
About Worfcforca Data Elements 





Stored 


Easily 

retrieved I 


IpdatMl 


Education 


Years of schooling 


8 


5 


6- 


Degree(s) awarded 


8 


6 


6 


Date of degree(s) 


8 


6 


6 


Major tidid of study 


8 


5 


6 


Institution attended 


5 


2 


1 


Grade-point average 


1 


1 


0 


Rank in class 


0 






College entrance test sco'^es 


0 






Other test scores 


0 






Continuing education, training 


Source (internal, external) 


7 


6 


6 


Duration 


8 


7^ 


7 


Course name 


7 


6 


7 


ProfMSlonal certificates, licenses 


Certification, licensure record 


4 


2 


3 


Examination scores (CPA, bar) 


2 


0 


1 


Wdric experience 


Generai work experience 


6 


2 


3 


Specialized work experience 


6 


2 


3 


Promotion history 


8 




8 


Step increase 


8 


6 


8 


Removal actions 


8 


7^ 


8 


Awards (monetary) 


8 


6 


8 


Awarc's (other) 


7 




6 


Performance appraisal score 


8 


0^ 


8 


Other managerial ratings 


3 


2 


2 



*The first four education data elements ara updated tncons/stentty 

^^nd'cates the four nDost commonly retnevabte items, which are dfscussed m the text 

below, since 1982 opm only requires agencies to submit aggregated data 
on training rather than individual-level data.) Two common automated 
items of possible relevance concern promotions and removals. Tlie 
fourth common element is a single digit that is the summary scx)re from 
the agencies' annual performance appraisal. Because rating systems 
vary and supervisors' standards for making judgments can change from 
time to time, this data element is not very useful a-s a generai indicator 
of quality. 
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Some items listed as **stored'' are found in the paper records in individu- 
als' official personnel files, which are decentralized in each agency (in 
up to hundreds of separate personnel offices per agency, around the 
country or overseas) and hence dif Hcult to use. Further, few elements 
meet a criterion of recency; agencies indicated that only the annual per- 
formance appraisal score and the payroll-related items such as promo- 
tions, step increases, and cash awards were automatically and re^nilarly 
updated. Agency officials explained that education data elements are 
not generally updated; updating occurs only at employee initiative or at 
the time an employee files a new application form when changing jobs. 
(See table 3.1.) 



Data on Attitudes, Values, Employee attitudes form the second of the three parts of the definition 
and Motivation workforce quality. None of the eight agencies reported that recent 

data from broad employee sur eys of this type were included in accessi- 
ble record systems. Several agencies reported occasionally gathering 
such data from targeted groups such as managers or workers in a partic- 
ular unit. Three agencies reported large-scale surveys, but only two of 
those had been done more than once, and the data were not accessible in 
any case. 



Data on the Match of Agencies match individuals and jobs routinely as part of staffing. How- 

Individual Capabilities and information about individuals' degree of cai)ability or the con- 

the Needs of the Job sistency of such capabilities with the needs of the job was reported to be 

part of accessible record systems by any of the eight agencies surveyed. 



Other Information We also asked agencies if speciaJ workforce quality data were kept on 1 1 

specific professional and administrative jobs or whether other indica- 
tors of quality were included in the accessible records beyond ones 
already probed in our survey. With the exception of Air Force and Navy 
which record some details of employee*^' job assignments, the agencies 
responded negatively to both of these questions. We asked whether data 
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were kept on groups of job applicants, to check the possibility of 
analyzing characteristics of those seeking federal jobs. None of the 
agencies had systematic information of this kind. In responding to 
our survey, agencies reviewed their data held at all levels, including 
headquarters and bureaus, on the chance that quality measurement 
might be advanced in particular sub-units; several examples came to 
light, reported in the next section. 



The data on existing indicators show that agency practice is diverse con- 
cerning the automation of records and that more data are stored than 
are now automated. In addition, the eight agencies reported idiosyn* 
cratic examples of other approaches towards assessment of workforce 
quality, such as special data items or data sources used by sub-units. For 
example: 

The Department of Justice analyzed applicants for attorney positions by 
requesting individuals' self-reported law school class rank and any 
clerkship ix)sition (which was evaluated for the prestige or hierarchical 
level of the court;. The data were interpreted as showing the overall 
quality of applicants, as well as differences in relative quality of appli- 
cant groups from different law schools in any particular year. The dat?. 
are not computerized. 

The Environmental Protection Agency maintained special preapplication 
files on high-quality potential employees in shortage occupations with 
information gained from recruitment contacts on indicators such as 
grade-point average and skills. 

The U.S. Geological Survey in the Department of the Interior collected 
aimurl updated professional resumes from physical scientists who 
chiefly do research and formally evaluated them using a structured 
method. The data are not computerized. 

The Air Force and Navy code the "skills" of employees as part of the 
automated record system. (These codes describe details or specializa- 
tions within the formal job description, for example, that a nurse is 
assigned to assist in surgery, rather than evaluations or measures of 
actual skills possessed by an individual.) Air Force (but not Navy) 
searches the information to automatically generate lists of candidates 
for vacant positions by matching skill codes in the vacancy notice with 
skill codes of those eligible to be considered. 



Feasibility of 
Enlarging 

Administrative Data 
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The inconsistency and incompleteness of the current automated agency- 
wide data systems and the individual efforts discussed above both sug* 
gest the idea of enlargement of the data bases so that more diverse 
indicators can be more consistently accessible on a wider population. 
This might be done by a combination of automating data from the 
existing paper record or augmenting the source documents and including 
new data in the automated files. The eight agencies provided informa- 
tion and views on this idea. 

Although file space is technically available on at least 7 of the agenc> 
data systems, the agencies reported several other potential problems, as 
shown in table 3.2. Computer programming was most frequently cited as 
a barrier (by 5 of the 8 agencies) because the older systems in use do not 
emplov up-to-date data base management methods permitting rs^id 
changes. Related problems of editing, testing, and debugging of new pro- 
grams, needed because of the complexity of changing many instructions, 
were cited by 3 of the 8. The same number of agencies also reported that 
there could be a long waiting time for the actual development work, 
owing to backlogs of planned system enhancements. Other problems 
cited less frequently but still of potential impact in a wider group of 
agencies, included implementation problems at highly decentralized data 
entry points and missing system documentation that impedes some 
kinds of changes. (See table 3.2.) 



Tabte 3^: Number of Agencies Citing 
Barriers to Enlarging Automated 
Personnel Data Systems 





Number of 


Barrier 


agencies 


Extensiveness of necessary programming 


5 


Extensiveness of editing, testing, debugging programs 


3 



Backlog of system enhancements (long waiting time) 3 



Time estimates for preparatory work to add new data elements to the 
automated systems varied too widely to be interpretable, as did cost 
estimates, and three agencies did not provide either estimate. Time esti- 
mates ranged from one month for a single addition to 6-8 months. Dollar 
cost estimates ranged from as low as a few himdred dollars to as high as 
$75,000, despite our efforts to specify the cost elements involved and 
the scope of the change. Standardizing the estimates by the size of 
employee population involved yielded i)er-i)erson costs of the change 
ranging from $0.01 to $3.33. None of the agencies estimated the addi- 
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tional costs of the time of data entry operators who would need to key 
information into new data fields in a modified system. 

Officials in charge of personnel data at six of the agencies offered oev 
eral views about the overall 'X)ncept of assessing quality by monitoring 
data drawn from administrative records, as outlined in table 3.3. The 
overall cost was a concern at three agencies; the likely accuracy of the 
data and the potential for misinterpretation of a relatively limited data 
set troubled three as well; two mentioned difficulties in gaining agencj' 
acceptance of a mandated and burdensome requirement of uncertain 
local use; and two feared grievances from employees. (See table 3.3.) 

Table 3^: Number of Agencies Citing 
Specific Concerns About Enlarged Data 
Systems for Rffonitoring Quality 





Number of 
agencies 


Concern 


Potentially high extra costs 


3 


Inaccurate data, misinterpretation 


3 


Lack of user acceptance, irrelevance 


2 



Possibility of employee grievances 2 



Data That Are 
Available in 0PM 
Record Systems 



The source of almost all opm data is the agencies. Therefore, opm has 
little additional information beyond that reported in the preceding sec- 
tion of this chapter. We examined the major opm record system, the Cen- 
tral Personnel Data File, which includes 55 data elements on each of 
over 2 million federal civilian employees. Each employee's file contains 
42 data elements submitted by the agencies and 13 generated by the sys- 
tem; as with agency data, the cpdf elements primarily reflect employees* 
current status on items related to pay and benefits. The education items, 
drawn from agency submissions, share the problems of limited overall 
comprehensiveness and inconsistent updating of the few elements that 
are on the file, opm officials believe little can be inferred from these data 
elements. 

We found several other opm data-collection efforts, but they are inher- 
ently limited or have been modified or discontinued so that they offer 
little at present. These include the following: 

Training data on individuals used to be part of the cpdf, but officials at 
OPM said they were dropped when budgets were reduced in the early 
1980s because there were few users of the data. Information on training 
is now collected from agencies and entered in a separate record system 
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only in aggregated form. It is difficult to interpret because a key item 
not collected is the course title. 

A government-wide attitude survey begun to aid in evaluating the Civil 
Service Reform Act of 1978 has not been re-administered since the third 
round in the early 1980s. 

The Professional and Administrative Career Examination was discontin- 
ued as part of the settlement of a court challenge of its validity for selec- 
tion. OPM officials could not recall any analyses of score trends in the 
overall tested population during the period it was in use, which might 
have been usefid if all or parts of the test could be adminisi'^ered to new 
samples for assessment purposes. 

No archive is tept of results of opm evaluation of various data on appli- 
cants (education, work experience, etc.) in those occupations for which 
central registers are still maintained, so no trend analysis of applicants 
is possible (except in the small number of job series where alternative 
tests have been developed). 



Modifications in OPM ^ statutory direction for the nature of personnel data 

p. . that agencies must report to the central file and opm officials believe 

there is little effective sanction for agency non-compliance with data 
requests other than the hypothetical withdrawal of appointing author- 
ity. Thus the cpdf is only as complex as permitted by voluntary coopera- 
tion, user needs, and available funds. The most recent overall review of 
the substantive content of the file was about ten years ago. opm officials 
said that over 400 suggestions for augmentation were reviewed, but 
only 15 were adopted. (A significant redesign completed in 1987 
improved technical aspects of the cpdf.) 



According to opm officials, any proposed new data items for the cpdf are 
reviewed on criteria of legal authority, cost, collectability, privacy con- 
siderations, and potential utility. Even when proposals are approved fol- 
lowing opm and agency review, long lead times are to be expected as 
agencies alter their systems at all levels as needed for the diange; basic 
compliance is piecemeal and data quality is expected to be poor for the 
first few years.2 
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0PM of flcials encourage alternatives such as sample surveys rather than 
the complex process of changing cpdf. Various offices within opm are 
msyo/ users of cpdf and are apparently satisfied. Individual agencies 
have more extensive data on their own employees and apparently rarely 
need comparative data. Although there may have been scattered 
requests, opm officials familiar with cpdf did not recall any steady 
demand from the msyor current users for enhanced information on 
workforce quality. 



Review of data now kept at eight agencies showed that information on 
indicators of workforce quality is limited in comprehensiveness, accessi- 
bility, and recency. A few measures of education are maintained in cen- 
tral computer fUes, but these are not consistently updated. Other 
indicators of education or work experience are available only on paper 
records in employees' official personnel folders, which are kept at 
decentralized personnel offices. Nothing on attitudes or job-skill match 
appeared consistently or in accessible form. The bulk of data that are 
accessible and up-to-date pertain to pay and benefits, which do not 
match the items needed for the proposed definition of workforce 
quality. 

Personnel data managers at the eight agencies raised concerns about 
cost, feasibility, and user acceptance in discussing the possibility of 
enlarging the existing data systems to include data either from existing 
paper records or from new sources. Officials were unable to provide con- 
sistent and comparable cost estimates for system modifications, but 
cited backlogs of needed system changes and technical problems associ- 
ated with any type of change to the common older computer hardware 
and software in use in the personnel record systems. These officials 
doubted that the potential use of data on workforce quality would be 
vigorous enough to assure acceptance of mandates to enlarge the 
systems. 

Data in opm's m^or system, the cpdf, include a small number of relevant 
items (on education) but they are derived from those in the agency sys- 
tems and are not up-to-date. There are few other kinds of usable infor- 
mation at OPM. Chances of change there appear slim as well. 

Since existing sources are inadequate and modifications unlikely, assess- 
ment of workforce quality must rely on alternative data sources. Both 
OPM and agency officials believe sample surveys could be an effective 
alternative. q o 
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The next three chapters outline our proposal for an ongoing program of 
assessment of workforce quality. The overall question of quality is to be 
addressed by a repeated assessment of the current workforce, at inter- 
vals, to assess changes, which will be supplemented by periodic studies 
of those entering and leaving the workforce. For each of these three 
approaches to studying quality we propose new data-gathering by sam- 
ple surveys; the three chapters discuss the details of our proposal in 
each case. This chapter presents the initial segment on assessing the 
entering workforce. 

New staff serve to replenish the lifeblood of an organization and in fis- 
cal year 1987 the federal government hired 40,573 new people for full- 
time permanent civilian professional and administrative jobs. Assess- 
ment of the quality of workers entering the federal workforce is of great 
interest forjudging whether the capabilities available for federal work 
are being adequately kept up and for use with other data in reviewing 
policies such as pay and benefits or processes such as recruitment and 
selection. Questions in this area are increasingly important as profes- 
sional and administrative jobs expand in significance in the federal sec- 
tor, wiiich places government more and more in competition with others 
for employees with advanced knowledge and skill. 

This chapter outlines m£yor questions to be answered about the entering 
workforce, evaluates alternative sources of data, and proposes a basic 
design. 



Questions About the 
Entering Workforce 



Any assessment in this area must answer two basic questions: How good 
are those attracted to federal jobs and how do they compare with others 
hired elsewhere? Within the framework of this general question, how- 
ever, it is important to draw the proper specific comparison. The stereo- 
typed view held by managers and co-workers is that newcomers at any 
particular time are rarely "as good' as those who are in or are leaving 
federal jobs. But such a comparison of entering versus current or exiting 
employees is inappropriate since by definition the newcomers come 
fr(Mn a different labor market, may have been selected using different 
criteria, and will always lack the old-timers' federal job experience. If 
there were data on newcomers at a series of points going back in time, 
today's entering workforce could be contrasted properly, not with 
today's experienced workers but with yesterday's newcomers. By start- 
ing the proposed assessment of incoming employees in comparable fed- 
eral and nonfederal jobs and repeating it, such a series will be 
established. 
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The question is not only the general quality of new federal employees, 
but the quality of the workforce in specific jobs. Although some policies, 
such as pay and benefits, affect federal jobs in general, most discussions 
of shortages of needed employee skills, most practical remedial actions, 
and even the special pay rate process that determines the need for 
higher salaries in areas of shortage, focus on specific occupations. 
Therefore data to answer questions about quality need to be at the level 
of specific occupational series. 

Some who reviewed our proposed design believed that the question 
should be refined even further, to ask whether there are differences in 
the quality of those entering an occupation in different agencies. \ Aft 
rationale is that for specific occupations, such as attorney or account- 
ant, agencies differ in the type and conditions of wck, the labor market 
conditions they face in recruitment and retention, and the efforc made to 
iv^Krruit a strong workforce. Consequently, it is argued, any assessment 
should provide data that could reflect the results of such differences in 
the actual quality of each agency's new hires. The analyris is plausible 
and the question is appropriate, but answering it may be technically dif- 
ficult. Similar questions could be raised about agency differences in 
overall workforce quality or in separated employees. Several general 
problems in reaching the goal of agency-level analyses of workforce 
quality are discussed in appendix III. 



Data Needed to 
Answer the Questions 
and Potential Sources 



The general problem for designing any data-gathering effort is how to 
provide su fficiently interpretable information in a feasible, timely, and 
affordable fashion, A range of sources can be considered, each with its 
own strengths and weaknesses. 

The definition of quality for the proposed assessment specifies two 
broad areas of information, individual capabilities and the relation of 
the individual's degree of capability to the needs of a specific job. 
Though new employees should be matched to their work just as should 
any other employees, given the way we are proposing to measure that 
match, strong d?it,a may not be available early in an employee's career. 
That is, because new employees are relatively unfamiliar wit the work, 
data on their perceptions of the match of their skills to the nee^s of that 
work could be unreliable. The supervisor would give a more informed 
view, but depending on the employee's time on the job, even the supervi- 
sor may not have had the time to observe a fair test of the match. We 
are proposing, therefore, that the assessment can be restricted to the. 
first part of the definition, so that information about new employees is 



41 



Page 40 



GAO/PEMIK88 27 A Framework For Stadying the Federal Workforce 



Chjipter4 

Assessing the Entering Workforce 



needed only on indicators of individual capability. (The second part of 
the definition is emphasized more iu the next chapter, in the assessment 
of the overall workforce.) For individuals new to the workforce as well 
as the job, capability can be assessed chiefly through educational indica- 
tors. Some will have experience as well, especially those hired at higher 
levels. 

Because the information is needed not only on federal employees newly 
hired, but on comparison groups of others hired for similar jobs else- 
where at roughly the same time, the question of data sources is complex. 
There is no source of comprehensive information on the characteristics 
of people in specific occupations and gathering new data presents signif- 
icant sampling challenges. 

The 3X3 matrix in table 4.1 summarizes our assessment of the per- 
formance of several sources of data on employees* education and exi)eri- 
ence with regard to feasibility and interpretability. We define feasibility 
as the degree of technical effort and cost needed for such tasks as draw- 
ing samples, obtaining access to existing data, or collecting new data. 
Interpretability refers to the clarity of inferences that can be drawn 
from the data on the questions of workforce quality. (See table 4.1.) 











Table 4.1: The Feasibility and Interpretability of Altematlve Data Sources for Aeeesslng New Employees 






InterDretabllity 




Feasibility 


Low 


Medium 


High 


Low 






School records on graduates' 
achievement and employment 

Employers' files on employees 


Medium 


Employers' opinions on education 
and work background of employees 


Employees' self-reports (nonfederal) 




High 


School faculty or administrators' 
opinions on graduates' achievement 
and employment 


Employees' self-reports (federal) 
College entrance test scores 





The diagonal pattern in the table suggests the general conclusion that 
the easier information to obtain is less useful and the most interpretable 
data are the most difficult to obtain. For example, the upper right seg- 
ment of the table shows a potentially useful source that we found is not 
available. If schools maintained records that would allow linking stu- 
dents' school achievement and their later job (both occupation and 
employer), a sample of graduates could be drawn, the records could be 
searched to extract these data, and analysis would involve comparing 
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those in federal jobs with others. However, when we discussed their 
records with officials at a number of colleges and universities, few had 
systematic information on all types of graduates (some schools track 
some kinds of graduates) and academic achievement records are main- 
tained separately from graduates' placement records. 

New employees could be located at their work, where employer records 
such as job applications, interviews, or tests, might provide the need(?d 
data on employee education and experience. However, the extensive 
information submitted by federal job applicants (on a common applica- 
tion form known as standard form 171) is not entered into accessible 
computer files but kept in the official personnel folder, so that extrac- 
tion of the data for a sample would require searching in hundreds of 
scattered personnel offices. (Federal hiring is increasingly decentralized; 
for the small number of occupations for which central hiring registers 
are maintained, data resulting from the centralized evaluation of appli- 
cants is discarded routinely.) Access to employer files outside the fed- 
eral government is uncertain and the information would vary in extent, 
comprehensiveness, and format. 

In contrast to the difficulties of locating these data from formal records 
on individuals, it would be much easier to gather opinion survey data in 
which knowledgeable people characterize groups. For example, either at 
schools or work places, officials (placement officers, faculty, personnel 
specialists, general managers) could provide views on characteristics 
such as education, work experience, and attitudes of people entering 
various occupations in federal and nonfederal settings. Indeed, this has 
been done several times recently with federal managers and personnel 
officials, as cited in chapter 1, and the Merit Systems Protection Board 
surveyed higher education officials' views several years ago also. Even 
students themselves, before choosing jobs, could provide opinions about 
their characteristics and how they seem to be matched to different 
places of work. However, no matter how systematically the opinion sur- 
vey might be done, the resulting data would be hard to interpret. Noth- 
ing new would be provided by repeating such surveys and our objective 
is to go beyond the shortcomings of such opinion data. 

Intermediate in interpretability are two kinds of data that we propose 
for use here, self-reports and college entrance test scores; both are quite 
feasible. First, individuals new to their jobs (both federal and other jobs) 
could provide data on their education and work background. Direct 
questioning of individuals avoids the problems of access and compara- 
bility that arise in using existing records. The data obtained from self- 
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reports can be inaccurate (from weak memory or deliberate distortion), 
but the amount of bias may be small since the self-report is not for deci- 
sions but for research. In any case, the self-reported data for a sub-sam- 
ple would be checked against other sources at an early point in the 
assessment project to estimate the degree of bias. School attended tells 
little by itself, but can be interpreted as a secondary indicator about the 
individual by use of a school quality measure.^ Feasibility is high for 
locating new federal employees to gather the needed data, but locating 
samples of nonfederal employees in speciflc occupations is a challenge. 

National standardized college entrance test scores are also of only mod- 
erate interpretability, shown by the long history of controversy about 
their meaning, though they continue to be used as aggregate indicators 
of educational outcomes.^ However, we determined that it is feasible to 
obtain thp scores by contracting with college entrance testing organiza- 
tions lor omputer matching. Upon official approval by the authorities 
who control the examination data on behalf of the original students, the 
firms call extract from their historical files any scores for members of 
study sample? based on name, date of birth, and social security number. 
To preserve a»nfidentiality, the data are returned for analysis in a way 
that keeps mdividuals' names and scores from being associated. For fed- 
eral professional jobs for which there are specific educational require- 
ments and most new employees have college degrees, such scores would 
probably be on file for most of those in any sample; the same is true for 
any nonfederal group as well. For the group entering administrative jobs 
Icr ivhich college is not reqi'ired there would be some missing data. The 
scores from the two different tests cannot be combined. The feasibility 
of obtaining and interpreting other standardized test data in specific 
(for example, tests in fields such as law, medicine, nursing, or 
r^r:^ ;ntin^») can be explored when specific job series are chosen for 
assessment. 



Proposed Desifin ^^^^ design is to survey samples of recently hired federal workers 

^ in selected occupations and comparison groups of nonfederal workers 



'.TVre are several existing rankings of higher education institutions or units within institutions, 
^ using measures from various sources (empirical data or informed judgments) on f iculty and students. 
See David S. Webster, Academic Quality Rankings of Amencan Colleges and Universities (Springfield, 
111.. Charles C. Thomas, 1986). 

^The Department of Education gives nationwide prominence to statewide average scores on the two 
commcn college entrance tests (the Scholastic Apiitude Test of the College Entrance Examination 
Board and the exam set by the Amencan College Testiiig Program) in an annual "wall chart" of 
educational indicators. 
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recently hired in similar jobs. The next sections discuss the occupations 
which could be selected, the sampling approaches, and the indicators on 
which data would be gathered. 



Occupations to Be Selected This assessment should focus on occupations for which entry quality is 

of particular interest. Various criteria could be used to select jobs for 
study, such as unusual losses of experienced employees, particularly 
acute pay disparities that might have caused lowered recruitment sue* 
cess, or anticipated changes of mission that are believed likely to 
increase the importance of the quality of employees being hired. Assess- 
ing a number of occupations at once allows for efficiencies at all stages 
of the work, though there is nothing particular about measurement or 
analysis that dictates how many occupations are assessed or when. For 
purposes of estimating the overall costs of an initial assessment, we 
have assumed that 12 occupations would be selected and that samples 
of federal and nonfederal employees in all of them would be surveyed at 
the same time (a total sample size of about 4,800). This arrangement 
would permit analysis of a sizable body of information, which seems 
preferable since there is likely to be great interest in the results. The 
alternative of spreading occupations across several years would delay 
providing the full results. Api)endix III gives further details of the sam- 
ple size and costs under these assumptions.^ 



Sampling of New Federal a sample of newly-hired federal employees in each occupational series 
Employees interest can be readily drawn using the Central Personnel Data File 

and surveys can be distributed to these individuals through their per- 
sonnel office, which is also shown on the cpdf record. We discussed with 
agency officials the feasibility of reaching people sampled in this way, 
since agencies have experienced the method as used in surveys n om opm 
and MSPB in recent years. Although some degree of burden is inevitable, 
the method poses no special difficulties. 



four-year interval between measurement cycles suggested for the core assessment may be rea- 
sonable here as well, except that places a large work load on the staff to mount both at the same time. 
Some key occupatk>ns should be assessed again, especially if policy changes have affected them. But 
others can be added 
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Sampling of the 
Nonfederal C!omparison 
Group 



We located two approaches to sampling the nonfederal group, through 
employers or through higher education institutions/ The first way is to 
use lists of employers developed as part of state unemployment insur- 
ance record systems. From these, and information on where specific 
occupations are concentrated in different types of firms, it would be 
possible to sample employers likely to have hired people in jobs compar- 
able to the chosen federal occupational series.^ The numbers of new 
hires will vary greatly by employer so sampling may be necessary 
within some firms. In general, we conceived that the next step would 
take one of three forms. Employers could provide lists of all newly-hired 
people in the occupations for the assessment staff to use in sending out 
surveys (to all or a sample). Alternatively, the employer could distribute 
surveys directly to selected new employees (to be returned directly to 
the federal agency administering the assessment). In a third alternative, 
the employer could extract data from individual employees' files. 

We attempted to more precisely evaluate the feasibility and costs of 
these alternatives with experts in the Bureau of Labor Statistics (bls) 
who work with states in surveying employers and who also use a 
national merged employer list routinely for gathering aggregated data 
on numbers of people working in specific jobs. We learned that bls has 
no experience using the employer lists to try to contact or gather data on 
individuals as our design requires. Thus bls officials could not evaluate 
the feasibility or cost of obtaining the new-employee information. A 
pilot test of one or more of the three approaches would, however, 
answer such questions readily. For efficiency, this methodology could be 
attempted in only a few states. These could be chosen on criteria such as 
high existing concentration of federal employees in the target occupa- 
tions or exceptionally keen competition for entry-level employees in the 
selected occupations. Though its feasibility is not clear, the employer 
approach appears to deserve further study because it would yield 
results on a generalizable population of new hires at all levels of experi- 
ence working for a wide variety of employers including state and local 
government, nonprofit institutions, as well as private firms 



^See appendix HI for more details of this sampling problem. 

An important criterion is similarity of woiic, since proper interpretation of data on new employees' 
capability requires that work in federal and non-fedf jobs with the same title be equivalent. The 
comparability of work under various systems of job classification has been examined in detail by the 
Bureau of Labor Statistics and 0PM to produce "cross-v^aiks" from the more general Dictionary of 
OccupatkMial Titles to the umque job titles in the classifkratk>n system used in the Federal Personnel 
Manual. 
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Because of the uncertainty of locating new nonfederal employees via 
their employers, we examined a second approach involving surveying 
recent college graduates sampled and located from school records. We 
found the Department of Education periodicaUy conducted a Recent Col- 
lege Graduates Survey that sampled those receiving either bachelor's or 
master's degrees and questioned them approximately one year after the 
degree date. We examined whether this survey, with modification, could 
be used for the proposed quality assessment, but as we finished our 
work the department decided not to continue this survey. The depart- 
ment is planning a more complex longitudinal survey of college students 
to begin in 1990, which could also serve our purposes. Appen^iix III 
gives further details of our review of the prior survey and the issues 
that would need to be resolved if the planned future survey is similar. 
(The main concerns include augmenting the survey questions so that 
data are gathered on our quality indicators including detaiki of educa- 
tion and work experience and assuring that enough people /ire located in 
the specific occupations selected for assessment.) 

There are two drawbacks to a design based on using any survey of 
recent graduates. First, it provides a comparison group only of those 
entering jobs shortly after obtaining a degree. The federal employee 
sample could be made comparable, but the assessment in that case 
would not provide data on qualities of more experienced workers enter- 
ing the federal and nonfederal sector?. Second, for those jobs where a 
degree is not required (such as all the federal jobs categorized as admin- 
istrative), having a comparison group of degree-holders only will over- 
state the education levels in comparable nonfederal jobs. Nevertheless, 
this approach can be explored further, as it appears to require only 
modest modifications to a planned survey, if the preferable alternative 
approach through employers turns out to be impractical. 



Indicators Table 4.2 shows the data elements from the general set that are needed 

for this particular assessment. All but the test scores would be gathered 
by surveys: of individual employees in the case of the federal workforce 
and of either individual employees or their employers in the case of the 
nonfederal workforce comparison group (as discussed above and in 
appendix III). (See table 4.2.) 
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Tabto 4.2: DimmsiOM, Indicatort, and 
Data SourcM for AsMSSing Naw 
Employaas 



Dbnanaion and indicators 



Knowledga, akiU, ability 



Education 



Years of schooling 



Degree(s) awarded 



Date of degree(s) 



Grade-point average 



Rank in class 



Institution attended 



Majof field of study 



College entrance test scores 



Other test scores 



Continuing education, training^ 



Quantity (hours, days, units, credits) 

Kind (course names) 

Source 



Professional certificates, licenses 



Examination scores (CPA. bar) 



Certification, licensure record 
Work experience* 



General work experience 



Specialized work experience 



Promotion history 



Awards (monetary and other) 
Attitudea, values, and motivation 



Matcii of individual capacltiaa and job neada 



Sourcaof data 



Employee survey 



Testing organizations 
(to be determined) 



Employee survey 



Employee survey 



Employee survey 



Employee survey 



Not Applicable 



^Applicable to recently-hired employees with prior work experience 

The table lists 9 education indicators; all are conimonly used to charac- 
terize the kind, quantity, or quality of schooling, but the list is almost 
certainly redundant to some degree. As experience with the assessment 
accunmlates the list can be refined and the best data items retained. The 
category of other tests could include, among others, graduate entrance 
examinations of various kinds, such as the Law School Admissions Test; 
their accessibility and interpretability can be explored when specific 
occupations are selected for study. As the movement to measure out- 
comes in higher education grows, there may be more and more test 
scores on sizable populations, such as all graduates in large state sys- 
tems, that could be explored for this assessment. Work experience and 
continuing education are relevant indicators for those entering a new job 
with some background, but the feasibility of comparison data on 
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nonfederal new-hires of this type is somewhat uncertain, as discussed 
above. Performance appraisals of employees from a wide variety of pre- 
vious employers may be difficult to interpret, so we do not suggest these 
be collected. Extensive new attitude survey data generally are not pro- 
posed for this assessment, but a few questions could be included in the 
employee surveys, for example, to gather reasons for taking different 
jobs, evaluations of recruitment processes, or commitment to the 
employer. 

The analysis of the data would be straightforward, examining differ* 
ences between new federal and nonfederal employees (within the same 
occupation) in the degrees earned, grade point averages, class standing, 
test scores, institutional quality ratings, years of experience, and so 
forth. Major pitfalls in analyzing and using these data could be avoided 
by taking specific steps such as: 

• warning users of the importance of judging individuals on their own per- 
formance and not attributing to any individual the aggregate character- 
istics of a group; 

• checking results carefully for signi^cance; avoiding overemphasis on 
small differences in relatively crude measurements sucli as self-reported 
grade-point-averages; 

• releasing data on a wide range of indicators at once to allow the most 
balanced and comprehensive interpretation and to avoid undue public- 
ity for single indicators such as college entrance test score; and 

• avoiding any generalization beyond the specific occupational series 
selected for study. 



Probably the largest amount of discussion of the quality of the federal 
workforce centers on those being hired. For this group interpretable 
measures of quality are feasible to obtain, including education, work 
experience, and test scores. The critical analysis will be to compare 
characteristics of new federal and nonfederal employees in similar jobs. 
Some effort will be required to obtain the needed nonfederal samples 
and data. Indicators such as education and test scores do not provide 
direct judgments of how well people will do on the job, but they do pro- 
vide useful relative indicators about those entering federal and 
nonfederal jobs and can signal comparative problems that deserve closer 
study and action. 

With the recommended study design, data would be available for the 
Hrst time to help provide generalize^ a^wers to such questions as: 



Summary and 
Conclusions 
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• To what degree does the federal government attract new professional 
employees, such as lawyers and accountants, from highly-regarded col- 
leges and imiversities? 

• Do those entering federal contracting positions have less training and 
experience than their counterparts in private industry? 

• What is the academic standing of incoming federal professionals com- 
pared to that of people hired in other sectors and are there some occupa* 
tions that are better or worse than others in this respect? 

• In an occupation like nursing in which many different degrees and cre- 
dentials are possible, do those entering federal service have patterns of 
degrees and credentials different from those taking jobs elsewhere? 
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The federal workforce is a dynamic system in which change is continu- 
ous. But what are the cumulative effects of the myriad changes? On bal- 
ance, are employees better-educated now than earlier? Is their training 
equipping them to keep pice with their changing work? To highlight 
important trends such as tiiese, it is useful to freeze the system and 
draw a portrait, which can then be compared to portraits from the same 
vantage point at intervals later. Changes> should stand out clearly when 
the images are juxtaposed. Designing the database for this long-term 
analysis has been the main focus of our work. 

When considering the entire federal professional and administrative 
workforce, made up of people with a wide range of experience, ques- 
tions about quality should not focus on initial qualifications alone. 
Trends in education indicators such as those emphasized in the previous 
chapter may indicate short-term advantages or disadvantages of federal 
employers in the entry-level labor market. For this part of the assess- 
ment, however, the emphasis should be on broader questions of the 
extent of csq)acities developed in the workforce through all kinds of edu- 
cation, training, and experience. Further, we should try to find out 
whether such capacities are being properly applied to the work, so that 
the workforce is neither over- nor uncfer-quaUHed for its assignments. 

This chapter presents the design for the second proposed segment of the 
assessment, to provide a broad picture of the workforce, intended to be 
rei)eated at planned intervals. Sections of the chap'^er present mjor 
questions guiding the design, data needed, evaluation of alternative data 
sources, and speciHcs of the design including occupations to be studied, 
sampling, and indicators. 



Questions About the 
General Workforce 



The basic question here is how good is the workforce and how has that 
changed since the last look at it? The question can in fact be posed quite 
generally and a general answer could be meaningful with statements 
about the level of education that prevails in the workforce at a certain 
point, for instance. Because of the growing predominance of white-collar 
jobs in government (excluding the Postal Service), the federal workforce 
in recent years has had a higher education level than the overall 
workforce. A sizable number of occupations should be included so that 
such general statements are well-grounded. 



But it is more probable that questions will center on comparisons within 
specific occupations as well, so the design should not aim simply to be 
roughly representative of the professional and administrative group. 
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but must also be targeted on specific occupational series of interest. 
Scientists and engineers, for instance, will undoubtedly continue to be of 
interest. As in the first question, addressed in chapter 4, on the quality 
of the entering workforce, questions will also be asked about agency dif- 
ferences, which may be feasible to answer about some occupations but 
not others. So that the design can assure an initial consistent baseline of 
information across all the jobs examined, we suggest postponing 
attempts to provide agency-level conq)arisons, though after further 
technical development, that is a logical candidate for an expansion of 
the effort.' 

The question of quality in the current general workforce (as opposed to 
those just entering) does involve the second part of our overall defini- 
tion of quality. That is, it is important and logical here to inquire if the 
workforce is well-matched to its work. In addition, attitudes, values, and 
motivation toward the work can be examined, given that this population 
(again in contrast to newcomers) has a more reasonable basis for making 
statements on such subjects. 

Answers to questions about quality should take into consideration the 
employee's length of service, so that patterns can be identified in broad 
subgroups of the workforce who entered at the same periods, in addition 
to those in the same jobs. This distinction, pursued over time, allows 
answering questions, for instance, about whether the experienced 
workforce is as good as it was at earlier points in time. 



Data Needed to 
Answer the Questions 
and Potential Sources 



We considered a number of sources which might provide the data 
needed on the various segments of the definition of quality. The 3X3 
matrix in table 5.1 summarizes our assessment of the performance of 
several sources of data on the same criteria discussed in the previous 
chapter: feasibility of gathering the data and interpretability. (See table 
5.1.) 



*The main issues are whether, as the desired analyses expand, there are enou^ employees in the 
various categories in which comparison is wanted for usable samples to be drawn and the added cost 
of enlaiiging samples where feasible. Appendix III includes discussion of the problem of agency-level 
sampling and analysis. 
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FMtMWy 


Low 


Medium 


High 


Low 


Employee data in official records 


Behavioral simulations evaluated by 
experts 


Direct tests of employee knowledge, 
skill, and ability 

Direct observation of work 


Medium 


High 


Supervisors' opinion of employees' 
characteristics, attitudes 

Behavioral data, unobtrusive 
indicators (e g leave usage) 

College entrance test scores 


Employees' self-reports of 
characteristics, attitudes, and job 
match 

Supervisors' report on employee skill/ 
job match 





Direct measures (tests, observations, or simulation exercises) as dis- 
cussed in chapter 2 in general, are not feasible for the start of this 
assessment though they could be experimented with as development 
proceeds. If common data were gathered on employees' performance in 
standardized simulation exercises used, for instance, in various agency 
executive development programs covering common issues such as super- 
vision or group leadership, over time a sizable body of data would accu- 
mulate. Data on employees in their official files, as discussed in chapter 
3, have the disadvantages of being both limited and inaccessible. 

There are a number of feasible alternatives, which vary in usefulness. 
Lowest in interpretability would be data from surveys of supervisors' 
opinions about their employees' background, such as education. Supervi- 
sors are not useful sources of such factual data nor probably of informa- 
tion on employee attitudes. Employees leave traces in formal records, 
such as their use of leave. But the interpretation of such data in isola- 
tion is impossible. Scores on college entrance examinations taken years 
earlier, though easy to retrieve, offer little useful information on the 
general workforce. (We considered such scores somewhat more inter- 
pretable with respect to the entering workforce because of the lack of 
other indicf'.tors on that group and the likelihood that the tests had been 
taken more recently.) 

Useful and interpretable data could come from asking employees to 
report facts about their own background (education, work history, and 
so forth), to report their attitudes, and to evaluate the match of their 
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capacities to their work. Surveyed for research, outside of a formal deci- 
sion context, employees would probably give fairly accurate 
information. 

Because of the importance of the job-match aspect of the defmition in 
assessing the overall workforce, it is crucial to have a second viewpoint, 
that of the supervisor. Though this adds to the costs of the assessment 
by doubling the sample size in this segment, the added value of the addi- 
tional perspective, we feel, justifies it. 



Proposed Design 



Occupations to Be Selected The important criterion for judging the sample is coverage of diverse 

occupational series across the spectrum of professional and administra- 
tive work. The exact number is not critical. Generalization to very large 
populations (such as all professional employees) is not an objective; 
accordingly, statistical precision is only an issue within each occupa- 
tional series, where enough members should be sampled to provide con- 
clusions at an acceptable level of likely error. 

Review of the overall list of federal professional and administrative jobs 
showed that 23 occupational series included slightly more than half the 
total group. Appendix II gives more detail of the speciflc series we rec- 
ommend be included in the assessment. Two of the 23 are clusters of 
physical and biological science jobs. 

As repeated data collection is the requirement for trend analyses some 
core of occupations should remain stable from one assessment to the 
next. Additions and subtractions in the list, however, pose no problems. 
The assessment should be repeated every four years, to provide a data 
point between each presidential election. 



Sampling The sample of employees would be drawn from the master list in the 

CPDF which includes most (though not all) federal employees. Within 
each occupation, we suggest that the workforce be grouped into three 
clusters (0 to 3 years of experience, 4 to 10, and more than 10) and ran- 
dom samples drawn from each. These samples would be drawn every 
time the survey is administered, such as at the four-year intervals we 
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recommend. We considered the additional information that could be 
gained by using a panel design, in which an initial sample drawn for one 
assessment would be recontacted at the time of a second assessment. 
The strength of panel designs is that by tracing the same people across 
time they allow much better analysis of what may have caused changes 
in the group. We decided not to suggest a panel design, however, because 
we believe the assessment should begin by providing a set of descriptive 
data only. Even if we aimed for causal analysis, we would want to 
explore whether there may be more cost-effective alternatives, because 
of the complex logistics of maintaining the panel in a sampling design 
that has severr ; strata of occupations and levels of experience as ours 
does. Extensivo work would probably be needed to trace individuals and 
accurately replace those who leave the different segments of the 
sample. 

The overall sample would include about 27,600 employees and an equal 
number of super sors matched to each employee. (See appendix III for 
details conceiiung sample sizes.) The employee sample size is compar- 
able in scale to opm and mspb surveys of the workforce done in recent 
years. 

The CPDF record gives the employee's locatior ^n government, including 
the personnel office that maintains the official record. The supervisors 
for each employee sampled would be determined by the personnel 
offices Carvo>s would be sent to the appropriate personnel offices to be 
forwarded to the supervisors of record to be filled out on each of the 
sampled individuals. 

We do not propose a comparison grour of current employees in the pri- 
vate sector for this part of the assessment as we did in the previous 
chapter on entering employees. Our analysis of questions about the 
overall federal workforce showed that the more frequent concern is over 
changes within the federal group over time. In addition, as emphasized 
in chapter 4, there are significant problems of obtaining samples of 
nonfederal employees for new data-gathering, and we found no existing 
sources of data on our indicators of quality for a broad group of 
nonfederal occupations. 



Indicators Table 5.2 shows the data elements for this segment of the assessment 

ai^d the}'' sources. The employee and supervisor are the sources for all 
the new aata proposed. (See table 5.2.) 
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Dimension and indicators Source of data 

Knowledge, sicilf, ability 

Education 

Years of schooling Employee survey 

Degree(s) awarded 
Date of degree(s) 
Grade-point average 

Rank in class " " 

Institution attended " " 

Major field of study " " 

Continuing education, training 

Quantity (hours, days, units, credits) " " 

Kind (course names) 

Source " " 

Professi' 'ificates. licenses 

Exar.ii, scores (CPA, bar) " 

Certification, licensure record 
Work experience 

General work experience " 

Specialized work experience 

Promotion history 

Awards (monetary and other) 

Attitudes, values, and motivation Existing employee survey 

done by MSPB (with possible 
modification as needed) 

Match of individual capacities and job needs Employee self-appraisal 
Supervisory appraisaS 



As noted in chapter 4, the education indicators can be refined as experi- 
ence with their use accumulates. Work experience is an important indi- 
cator and experimentation will be needed to determine the best format 
for efficient surveying of needed information. Experience with the 
assessment will also suggest data elements of greatest power and ir^ 
est within the work experience area, among such asi)ects as the posi- 
tions held, moves made, rewards obtained, or others. There is very wide 
exi)erience in i)ersonnel with various ways of structuring questions on 
biographical items like these. Training is an important indicator for 
those who are some years beyond formal education. There are no central 
data gathered on individual federal employees* training at present. Thus 
some development will be needed to specify the questions in this area 
also so that employees can accurately report on the kind and extent of 
training and their achievement. 




Table S.2: Dimensions, Indicators, and 
Data Sources for Assessing the General 
Workforce 
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Attitudes and related views could certainly be gathered as part of the 
employee survey instrument although there is a sizable continuing 
assessment of federal employee attitudes which, with modification, may 
provide sufficient information. The Merit Systems Protection Board 
periodically conducts its Merit Principles Survey, most recently sam- 
pling 21,000 federal employees and asking a number of general ques- 
tions about morale and job satisfaction, among other subjects. There is 
no statutory requirement that such a survey be done regularly, so if it 
(or an augmented version) does seem usable in the context of the assess- 
ment of quality, it would be prudent to provide more assurance that the 
data series would continue. Because the previoiis mspb surveys have 
been designed for somewhat different purposes, one design issue would 
need to be reviewed and possibly some acljustments made. Past mspb 
employee samples have covered the entire workforce (not just profes- 
sional and administrative workers) and have allowed analysis by agency 
and also by grade levels government-wide, but not by occupation. Since 
occupation is one critical dimension in our design, some augmentation of 
the MSPB sample might be needed to insure adequate samples sizes for 
purposes of the proposed quality assessment. 

Because of plausible links of employee turnover and productivity to atti- 
tudes such as morale, job satisfaction, or commitment to the present job, 
measures of these employee views are sometimes discussed as part of 
analysis of the quality of the workforce. However, data in news articles 
or popular discussion may be based on poorly drafted questions and 
uxu^presentative samples. If such employee views are to be included in 
an assessment, /e believe they should be measured as regularly and 
soundly as possible and there is substantial experience with technical 
issues in the field to draw on both in social science attitude measure- 
ment and in public opinion polling. As indicated in our discussion of the 
definition of quality in chapter 2, this type of data requires cautious 
interpretation because of uncertainty about how the attitudes being 
measured are related to behavior. For example, does employee morale 
cause work effort or does it reflect external events unrelated to work? 
All measures of workforce attributes have a degree of uncertainty as to 
their link to other variables but we emphasize a special word of caution 
in this regard about the often-cited attitude variables. 



The match of qualifications and job can be exanuned through rating 
scales completed by both employee and supervisor. The instrument 
would differ from the familiar managerial appraisal forms by providing 
much more specific dimensions for rating, though it still need not be 
lengthy. Items could include some or all of those included in the 2-page 
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research performance appraisal instrument developed by opm and used 
as the dependent or criterion measure against which to test various 
predictors of work success. That instrument covered general dimensions 
such as the employee's work output, quality of work, accuracy of work, 
job knowledge, and diversity of tasks that can be done by the employee. 
It also requested supervisors* separate ratings of both the importance of 
specific abilities and the employee's level of each, such as oral expres- 
sion, written expression, reasoning, work relations with others, or 
adapting to changes in work. We believe the employee form should 
require substantiation of the self-assessment ratings (citations of work 
experiences or training, for example) to put some pressure on the 
employee for accurate responses, opm has experience with such instru- 
ments and has found the administration feasible and the results inter- 
pretable. Further technical development can determine the proper mix 
of general and specific abilities to be rated. 



Summary and 
Conclusions 



Useful data can be obtained to assess changes in the quality of the pro- 
fessional and administrative workforce. The two key dimensions include 
knowledge, skill, and ability; and the match of the individual's degree of 
capability to the needs of the job. Msyor data sources are the individual 
and the supervisor. A current government-wide employee survey done 
by MSPB may, with modification, serve the needs of the assessment for 
attitude and morale indicators. 



With proper design of repeated measures, data would be available for 
the first time to help provide generalized answers to questions such as: 

Are today's experienced employees in the selected attorney and 
accountant series as good as those of 4 or 8 (or more) years ago? 
Are we getting the skills we need i'^ the workforce of contract special- 
ists? Has that changed? 

How good are the scientists (in the selected series) who stay 10 years in 
government, compared to earlier times? 
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The other side of the coin from attracting a quality workforce is 
whether the best are retained. Retention needs to be long enough so that 
recruitment, orientation, and later training costs are recouped. It should 
also be targeted enough so that the most desirable employees stay on the 
job the longest. Those who leave may take with them leadership and 
management skills developed over long periods and especially suited to 
the special situations of federal programs. In areas in which government 
is working to develop greater expertise than it now has, such as use of 
computers, progress may be especially slowed if capable people are lost 
to other sectors. Where government may be in an adversarial role 
involving oversight or litigation with other sectors, in areas such as anti- 
trust, the environment, or contracting and procurement, loss of exper- 
tise to the other sectors can affect the balance in the relationship. Specu- 
lations and anecdotes about the quality of those leaving government are 
common, so that on ability to measure this aspect of changing quality 
also became an important focus for our work. 

This chapter completes the design of an overall framework for assessing 
the quality of the workforce by giving details of an approach to evaluat- 
ing the quality of those leaving federal jobs. The chapter has three sec- 
tions, following the outline of the two preceding design chapters: the 
first examines questions about the subject; the second reviews data 
needed to answer the questions and possible sources; the third gives 
details of the proposed design including approaches to selecting occupa- 
tions that might be studied, sampling considerations, and indicators. 



Questions About 
Those Leaving the 
Workforce 



The basic question here is whether the federal government is losing its 
best employees. If the answer is yes, that suggests a need to strengthen 
the incentives that maintain employees' commitment to their federal 
work. The overall assessment outlined in the preceding chapter is 
intended to answer general questions about the workforce across time 
and whether it is suited to its tasks. In a general sense, merely by exam- 
ining the trends in quality within the three levels of experience in the 
workforce sampled in such a study, the question of how well the govern- 
ment is holding on to quality employees could be tentatively answered. 
But trends in quality indicators in an aggregate group can result from 
both incoming employees as well as exiting ones, so that answer would 
not be as accurate or conclusive as might be wished about those being 
lost. More importantly, answers take a long time to develop in trend 
studies. The interval of observation we are suggesting is 4 years, so even 
if a m^uor baseline study were done immediately, the trend data would 
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Data Needed to 
Answer the Questions 
and Potential Sources 



not be available for about 5 years. Therefore, more immediate informa- 
tion would be useful, especially by providing details about the charac- 
teristics of those leaving that will not in any case be available from the 
core assessment, which examines quality only in the workforce 
remaining. 

To know whether separations include disproportionate numbers of the 
most highly desirable employees, it is necessary to compare those leav- 
ing and those staying, to see if the two groups differ in quality. Ques- 
tions focused on the first part of our definition of quality presented in 
chapter 2, concerning capabilities of individuals, are most relevant to 
this assessment. That is, it is of most interest to know if, on average, the 
separated employees are better than the rest in their degree of knowl- 
edge, skill, and ability or in the attitudes, values, and motivation they 
bring to their work. Since those leaving are no longer doing federal 
work, at least directly, it seems of less interest to attempt the assess- 
ment of whether their capabilities were, in the past, well matched to 
their job. We do, however, suggest that that be explored from the 
employee viewpoint as one of several opinion questions about perceived 
reasons for leaving government service. 



The question of capabilities lost when emnloyees leave government is 
basically the same question posed in the previous chapter about capabil- 
ities to be measured in the core assessment. The data needs and the 
alternative sources that could be explored for information about capa- 
bilities of all kinds are the same. Two workforces are to be contrasted, 
differing only in that one group is still in government and the other has 
recently departed from government. The amount of experience being 
lost is of interest and for those leading a federal job which is their first, 
educational indicators may also be of interest. 

The sources for these needed data on education and work background 
are perhaps more limited in this part of the assessment than in the pre- 
vious two parts. Those who have left by definition are no longer at work 
in federal positions and thus are not accessible on the job, so direct 
measures (tests, observations, ete.) would not be possible. A few data 
elements about promotion and award histories are available in the auto- 
mated records. But the remainder of the needed information on workers, 
such as education, job history, or training, would be to some degree sub- 
stantively incomplete, possibly out of date, and in any case inaccessible 
in paper records, as discussed in chapter 3, perhaps even more than 
usual since the separated emptoyees' files may be segregated in storage. 
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College entrance test scores could be obtained by the matching methods 
already outlined, though they would provide interpretable data about 
only a fraction of those leaving.* 

Employee (and former employee) self-reports, therefore, are once again 
the proposed source of the education, work history, and continuing edu- 
cation indicators. As discussed in both earlier chapters, supervisors are 
not a good source of information on employees' background or even 
their attitudes; more direct data are preferable. Employee self-reports 
are interpretable and are feasible to gather from current employees; 
locatmg separated employees (through the agencies) raises feasibility 
issues, but not impossible problems. How a sample of separated employ- 
ees would weigh the positive and negative incentives when deciding 
whether to respond can only be tested in the early stages of the assess- 
ment. Response rates of other recent assessments of separated employ- 
ees may not be generalizable since the surveys have been conducted on 
different populations such as the Senior Executive Service. 

Questior s about the views of those who left federal jobs cannot be 
answered by the attitude data from general surveys of current employ- 
ees discussed in the previous chapter. The final personnel action on an 
employee who is leaving is to include a reason, but this piece of data is 
hard to interpret because the employee may not be truthful. The answer 
might affect the chances of obtaining future recommendations from 
supervisors and does constitute the agencyV; official answer to inquiries 
from unemployment insurance officials, which can affect benefits. Thus, 
some attitude questions should be asked on the same employee survey 
we are propc ng be used to gather education and work experience data. 
Answers to these questions could give some understanding of positive 
and negative factors influencing the decision to leave. Further develop- 
ment work as thf; assessment matures can focus on exit interviewing or 
other data gathering approaches, so that employees will view the 
method as confidential and trustworthy and will therefore provide accu- 
rate information. These attitude data do not constitute a true causal 
study of why people left government; they are further indicators about 
the individuals, but a full exploration would require other data and 
measures to weigh a variety of possible explanations. 



'College admissions test data would be available on part of the experienced workforce. Files at the 
tcsttng firms go back about 23-29 years, so potentially there are scores on file for those under about 
age 38 in the case of the SAT, for example The more significant lunitation, however, is that test 
scores should be used as data only on those for whom few other sources exist, that is, those with little 
or no work experience. Smce the distnbution of professional and administrative emptoyees leaving 
government by age and work experience is not known, we could not estimate the size of the group for 
whom test scores mignt be available and mterpreUble. 
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Since the match of employees' capability and the needs of the job is not 
to be explored extensively in this part of the assessment, it is not neces- 
sary to ask current or separated employees those detailed questions or 
to survey supervisors to obtain a second evaluation. It might be desir- 
able to have some supervisory opinion data about separated employees, 
but there are several problems of feasibility and interpretability with a 
new survey of supervisors for the assessment. Simply locating former 
supervisors could be difficult, if some or all of the employee sample is 
drawn from a population leaving during several years, as the supervi- 
sors also could have changed jobs in that time. Supervisors' lack of clear 
memory is another difficulty. Coloration of the supervisor's report 
because of the circumstances of the person's leaving is a third.^ 

The 3X3 matrix in table 6. 1 summarizes our evaluation of the alterna- 
tive data sources. 



Tablo 6.1: The Feasibility and Intarpratability of Aitamative Data Sources for Assessing Separated and Retained Emplovees 






InterpretabUitv 




FeasUMHty 


Low 


Medium 


High 


Low 


Employees' data in official records 


Behavioral simulations evaluated by 
experts (impossible for former 
employees) 


Direct tests of employees' knowledge, 
skill, and ability 

Direct observation of work (impossible 
for former employees) 


Medium 


College entrance exam scores 

Reason for separation shown on final 
official record 


Former employees' self-reports of 
charactenstics. attitudes 




High 


Supervisors' opinion of employees' 
charactenstics and att'tudes 


Employees' self-reports of 
charactenstics and attitudes 





number of years of performance appraisal dat? arc available now in automated records at 0PM, 
but only on part of the workforce In the single-digit form in which it is stored, this indicator is 
ambiguous and we do not propose to use it elsewhere m the plans for assessmg workforce quality 
where we can obtam more mterpretable data on supervisors' views of the employee's abilities and 
match to the job. However, if further analysis shows that at least three years of appraisals were 
available on the type of employee samples to be drawn for this segment of the assessment, which 
would permit some analysis of trends in the performance of those who leave some might view tlrose 
data as a ufjeful addition 
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Proposed Design ^ 



Occupations to Be Selected As in the case of assessing quality in those entering the workforce, occu- 
pations should be selected for which possible losses of quality in the 
workforce are of particular interest. Unusual efforts by agencies in 
recruiting or retention may serve as a signal of occupations to be 
selected. Criteria for selection could emphasize: 

• occupational series in which large numbers or proportions are quitting 
(441 1 nurses left federal service in fiscal year 1987, for example, a rate 
of 12.1 percent, while 948 attorneys left, a rate of 6.7 percent); 

• occupational series in which losses of quality staff may hurt perform- 
ance of general services within government (such as computer special- 
ists, of whom 904 quit in fiscal year 1987, or contract specialists, which 
lost 6^2); 

• occupational series in which competition with the nonfederal sector is 
severe as shown by the size of the pay gap (such as high-level attorneys, 
engineers, and systems analysts, for which the 1986 pay comparability 
survey found the weighted median salaries of private sector workers 
were 30.7 per cent higher than those of federal GS-15 employees doing 
comparable work, $80,803 compared to $61,842). 

Assessing a number of occupations at once allows for efficiencies at all 
stages of the work, though there is nothing about measurement or anal- 
ysis that dictates how many occupations are assessed or when. For esti- 
mating costs, we have assumed that 12 occupations would be covered, 
all surveyed at once. 



The CPDF transaction files provide a source of all those leaving govern- 
ment. We suggest examining those leaving voluntarily; very few leave 
involuntarily. Within each occupation selected for study, a group of 
those who left can be sampled and the agency they left can be requested 
to provide the last known address. For mailing tax and other informa- 
tion, the address is usually maintained for at least a year. Further tech- 
nical development can explore the question of the exact definition of 
separation to be used in forming the group to sample. In addition to sam- 
pling those voluntarily leaving other than by retirement, there may be 
value in select^ing also from a group who appear to be retiring at an 
unusually early age. This discussion assumes that the full range of years 



Sampling 
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of service is examined. It would be possible tx) restrict the study to only 
those with substantial government service, if it were believed that qual- 
ity issues were most salient there. Loss of the most capable younger 
employees is of course also possible and quality indicators should be 
selected to provide interpretable data on all levels of experience 
included in any sample. 

The CPDF can be used to draw the comparison group samples, which 
would also be surveyed. These should be from continuing employees in 
the selected occupations, matched further in age, sex, and years of gov- 
ernment service, so that, as far as possible, demographically similar 
groups are being compared. 



Indicators The indicators of capability would be similar to those sought in the core 

assessment including information on education, especially later augmen- 
tations through training; licenses or certifications, especially if some 
fields have these at more advanced levels, indicating specializations or 
updated skills (such as re-certifications); and work experience including 
jobs, promotions, and awards. The list should be shortened as specific 
samples are determined; for example, grade-point average would be reli- 
ably recalled only by those recently in school and would be interpretable 
as an indicator of quality primarily for those with little work experi- 
ence. This item could be omitted if it is decided to focus on experienced 
employees only. 

Tn the area of attitudes, values, and motivation there is room for consid- 
erable development. Unlike the core assessment in which the goal is a 
general evaluation of quality and attitudes are of unknown relevance, 
with separated employees there is a specific event whose causes and 
consequences can be asked of the former employee. Thus a set of ques- 
tions can be asked that would explore the separated employees' work 
experience and values, what led to the decision to leave, and, if the per- 
son has a new job, what it is and what tlie individual finds more attrac- 
tive in the new situation. The answers will not be completely reliable, 
owing to the natural inclination to attribute favorable results to one*s 
decisions, but some useful generalizations can result about who is going 
where and what reasons appear to have motivated the moves. One such 
opinion would be roughly parallel to the job-match concept studied more 
directly in the core assessment. Sej)arated employees could be asked if 
their knowledge, skill, and ability are being used to a greater or lesser 
extent in the new job. Once attitude questions are designed for the sepa- 
rated group, similiar ones can be asked in any general employee attitude 
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survey (described in chapter 5) to provide direct comparisons to the 
overall workforce. Table 6.2 summarizes the indicators we believe are 
useful. 



Table 6.2: Dimentiont, Indicators, and 
Data Sources for Aaseaatng Separated 
and Retained Employees 



Dimension and Indicators 



Source of data 



Knowledge, skill, ability 



Education 



Years of schooling 



Employee survey 



Degree(s) awarded 



Date of degree(s) 



Grade-point average 



Rank in class 



Institution attended 



Major field of study 



Continuing education, training 



Quantity (hours, days, units, credits) 



Kind (course names) 



Source 



Professional certificates, licenses 



Examination scores (CPA, bar) 



Certification, licensure record 



Wbrk experience 



General work experience 



Specialized work experience 



Promotion history 



Awards (monetary and other) 



Attitudes, values, and motivation 



Employee survey (separated 
employees only) 



IMatch of Individual capacities and job needs 



Employee survey (separated 
employees only) 



SuminarV and useful data can be obtained on the quality of those leaving government 

p , . jobs. Quality would be defined in terms of employees' capabilities, 

IJOnCiUSlOnS including their knowledge, skill, and ability, and their attitudes and val- 

ues toward work. Data on those leaving would come from surveys of 
samples in selected occupatioa^ and would be compared to similar data 
obtained from employees still in the same federal occupations. 

D?ta gathered according to the proposed design would help provide gen- 
eialized answers to questions such as: 
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• When top scientists leave government, do they tend to continue doing 
govenunent-related work, although for a new employer, or are they lost 
to federal niissions? 

• What is the profile of the quality of attorneys leaving government, in 
terms of their legal education as well as their career paths once in gov- 
ernment? Where are they going and what do they say about their 
reasons? 

• Are the nurses leaving government service more educated on average 
thaiL ones who stay? 

• What is the average length of service of the most qualified people leav- 
ing government in these occupations — that is, do highly capable people 
more rx)mmonly leave at early or late points in tneir federal careers? 
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The assessment design described in chapters 4-6 is summarized in table 
7.1, which shows the three segments of the workforce proposed for 
evaluation and the relevant questions, data sources, and analyses that 
we have suggested for each. (See table 7.1.) 

Ensuring the adequate implementation of this design is as important as 
planning the study details. The appropriate structure is important 
because the task is complex, includes conceptual, methodological, and 
data problems, and will invoh^e some continuing federal funds, though 
not a large amount. Questions about implementation of the proposed 
assessment addressed in this chapter include: 

• What criteria should be used in considering the organizations within or 
outside the federal government that could be responsible for the 
assessment? 

• What funding issues should be considered as part of planning for the 
needed work? 

• Are there preferable approaches to timing and phasing segments of the 
assessment, if it cannot all be started at once? 

We conclude that a data base is feasible that can help answer the ques- 
tions raised in the committee*s request, and we recommend that the Con- 
gress should establish the assessment by direction to the executive 



branch. 




Tabto 7.1: DMign for an Assasamant of Fadaral Worfcforca Quality 


Sagment of worfcforca and partinant 
quaationa 


Indicators 


Data sources 


Entarlng worfcforca 

How good are those attracted to federal jobs 
and how do they compare with others hired 
elsewhere? 


Knowledge, skill, and ability (as indicated by 
education and work experience) 


Employee self-reports, test score records 


Current worfcforca 

How good IS the workforce and how has that 
changed over time'^ 


Knowledge, skill, and ability, attitudes, 
values, and motivation, match of capabilities 
to job needs 


Employee self-reports, existing attitude 
surveys (with modification), special 
supervisor appraisals 



SefMirations from the worfcforce 

Is the federal government losing its best Knowledge, skill, and ability, attitudes. Employee self-reports 

employees? values, and motivation 
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Diverse tasks are involved, including tasks of policy (deciding on the 
scope and objectives of the assessment), research and development 
(deciding on measurement instruments, sampling, pilot testing, refine- 
ment of measures in later cycles), logistics (contracting for survey sup- 
port, handling data tapes), analysis (interpreting the data), 
communication (reporting the results), and management and 
coordination. 

In addition, the assessment program should be located in a setting in 
which it can be seen not as a one-time project, but as a continuing activ- 
ity. This perspective may help to insure that trend analyses can be 
based on consistent measures and methods of interpretation that are 
now missing from the scattered discussions of workforce quality. A 
long-term perspective is also needed so that there will be support and 
direction for such additional activities as: 

doing research and development to refine the techniques used (as sug- 
gested in many places in the earlier chapters); 
reaching out to stimulate interest in agencies, policy research groups, 
and universities in strategic assessments of the workforce (comparing 
existing and needed quality); and 







Comparison 


Estimated sample sizes 


Frequency 


Analysis 


Entering nonfederal employees in 
similar occupations 


2,400 new federal employees, 
2,400 nonfederal (in 12 
occupations) 


As desired 


Within occupations, at one point in time, 
comparison of federal and nonfederal new 
hires on education and experience 


Federal employees in the same 
occupations at earlier times, 
supervisor and employee 
perceptions at the same time 


27,600 employees and their 
supervisors (in 23 occupations) 


4-year intervals 


Within occupations, across several points in 
time, trends in education and experience, 
attitudes, and degree of perceived match of 
employee cap2billies to individual jobs 


Nonseparated federal employees 
in the same occupations 


3,600 separated employees, 3,600 
matched nonseparated (in 12 
occupations) 


As desired 


Within occupations, at one point in time, 
comparison of those staying and those 
leaving on education, experience, and views 



Organizational 
Arrangements 
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• responding to those wishing to use the data including preparing public 
data tapes, or perhaps sponsoring secondary analy^^es that can extend 
the interpretation of the rich data that will accumulate. 

Criteria that should be considered in analyzing where to locate the 
administration of the assessment include: 

• existence of the required expertise and experience with the methods and 
data required for assessment; 

• access to data bases needed for sampling, comparison studies, etc.; 
- objectivity with respect to the outcomes of the assessment; 

• commitment to the effort and its stability over time (e.g. personnel, 
funding); and 

• availability of resources to carry out the tasks. 

We believe that such a continuing program of assessment of workforce 
characteristics properly belongs in the executive branch. Within the 
executive branch, opm has responsibility for workforce data, including 
the official i)ersonnel folders held at the agencies, opm maintains the 
most comprehensive automated workforce data base, the cpdf, which 
can be considered a subset of the folders' data, opm also has responsibil- 
ity for assisting agencies with obtaining the quality of staffing needed 
(including developing tests and other examination methods to determine 
quality) and for evaluating i)ersonnel management. This evaluation 
could include using, among other criteria, measures of the quality of 
workforce attracted and retained, opm is required to make several 
reports to the Ciongress on the workforce. As the agency with the most 
detailed knowledge of both the workforce and methods for assessing 
individual quai ies, opm is the logical candidate for performing the pro- 
po ;ed asse 3ment. 

Nevertheless, opm must be assessed on ine other criteria as well, opm 
might not be able to e fully objectiv? with respect to the outcome of the 
assessment, since quality problems reflected in the assessment data 
could reflect on staffing practices in which opm was involved or on other 
policies on wh-ch opm had advised officials of the executive branch. Fur- 
ther, OPM lost research capacity in the early 1980s, opm thus n;ay not 
have adequate resources to carry out the tasks and ma> not be able to 
mal.e the necessary commitment to a stable design and repeated assess- 
ments, though this could change in the f'^^ure. 

Relevant exj)ertise in examining labor market issues generally, including 
sampling and surveying employers, is available at the Bureau of Labor 
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Statistics. The bureau has dealt with issues of independence, timeliness, 
and technical quality in gathering and reporting economic indicators 
that are of intf^ns? r'blic interest. The subject matter of the workforce 
quality assessment may be more comprehensive (in the types of indica- 
tors involve i) than that of many projects in the bureau, however, and 
there are no current resources available for the task. 

Another possible organizational location for the assessment is the Merit 
Systems Protection Board, an independent agency created by the Civil 
Service Reform Act of 1978 to oversee and report on all aspects of the 
laws governing the federal workforce. In connection with its oversight 
of the personnel system and to provide data for a mandated report on 
significant actions of opm, the board has carried out a number of 
research and evaluation projects 'Concerning the federal workforce. The 
boaid by law is to be bi-partisan in coiriposition. The board staff has 
included experts in personnel studies and survey research in recent 
years, though fluctuating in extent, and there are no current resources 
for additional tasks. Specific direction to take on the new work and 
assignment of specific resources would be needed to insure that the 
assessment would be a stable part of the board's research and evalua- 
tion portfolio. 

Private organizations with experience in personnel research or general 
surveys are probably not suited to the multiple tasks involved in the 
overall assessment, though appropriate contracts for specific tasks 
could be used. Such organizations would lack ready access to federal 
agency records and data bases and would have weak influence over 
agencies' responsiveness to the needs of the assessment. 

An advisory board of individuals with experience and expertise in gath- 
ering and using data on workforces could be helpful to the oi:ganization 
eventually selected to perform the assessment. The group could include 
experts from various disciplines of data-gathering and analysis, as well 
as experiences^ personnel officials and managers from public and private 
sectors, along with individuals representing eventual users of the data 
from Congress, the press, and employee groups. 

An implementation structuie which should be considered would be to 
designate one agency as the responsible organization to carry out the 
assessment, but to mandate others as well to provide specific services, 
including bls for help with comparison groups of nonfederal workers, 
OPM for drawing statistical samples using the cpdf, mspb for the 
employee attitude data, and any-others that seem useful (Bureau of the 
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Census, Department of Education, etc.). Specific oversight roles for an 
external advisory board should also be designated from the outset, 
including review of designs, instruments, sampling and analysis plans, 
the resulting data and interp»retive products. 



Funding for the 
Assessment 



Although there are limits to what can be known about the quality of the 
federal workforce at the present time, with adequate funding and per- 
sonnel a responsible organization could do a good job in tracking indica- 
tors of quality for different segments of the professional and 
administrative group. If the assessment is to be done, however, new 
resources are needed, since at present no organization has a usable data 
base (as detailed in chapter 3) and no organization has such a task in its 
mandate or current plans. 



If the proposed assessment is established, specific resources should be 
authorized for a substantial term. The results of the assessment will be 
more useful if the study framework is stable, as that helps insure com- 
parable data across time, and such stability is uncertain without special 
precautions. If the government-wide employee attitude survey data are 
considered important, specific direction to mspb may oe needed as well, 
as there is no assurance that the board will be able to continue the sur- 
vey at regular intervals or will maintain stable methodologies which 
encourage valid comparisons (similar samples, question wording, etc.). 



So that the data from the assessment can be used by various analysts, 
additional funds could be useful for a program of independent spon- 
sored research using the database. Staff and resources could also be 
made available for synthesizing studies done by individual agencies that 
could provide benchmarks or comparisons to the assessment data or for 
smaller studies in a few agencies or specific occupations in which 
targets of opportunity arise. For example, once the instrument package 
is ready, the assessment instruments could be used to evaluate effects 
on the quality of the workforce resulting from personnel policy demon- 
stration projects such as the Navy's China Lake experiment. In this way, 
the assessment group could represent a body of knowledge and exper- 
tise to expand the effort to assess workforce quality beyond the assess- 
ment itself. Funds should in any case be provided to prepare 
documentation for public use computer tapes of th'^ assessment data, 
which can be deposited in the National Archives' collection of machine- 
readable data. 
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A detailed cost estimate for the assessment is included in appendix III, 
which shows that initial assessments of entering and exiting employees 
in selected occupations (together with necessary comparison groups), 
plus collecting the baseline data for the core assessment trend analyses, 
would involve an estimated direct federal cost of approximately 
$718,000. Some costs cannot be accurately estimated at this point and in 
particular the key comparison study of new-hires in nonfederal organi- 
zations will involve unknown but possibly substantial costs. The total 
costs, both new expenditures and the cost of employee time to complete 
survey instruments, appear modest, however, in comparison to the 
potential value of usable data on the quality of results obtained from the 
substantial time and funds spent in recruitment, selection, and other 
efforts to attract and retain a high quality federal workforce. 



Because of the widespread discussion of the quality of those in public 
service, the assessment should begin as soon as possible. In anticipation 
of the transition to a new administration in 1989, followed by possible 
changes in program policy and personnel policy, it would be important 
to gather baseline data quickly on the qusdity of those in the federal 
workforce now, before such changes take effect. 

Table 7.2 shows the relative complexity and difficulty of the three parts 
of the assessment, as outlined in chapters 4-6. Each poses distinctive 
challenges, including obtaining the comparison group of nonfederal new 
hires for the first segment, efficiently managing the overall scale of the 
stratified samples and multiple instruments to be used in the core 
assessment, and locatmg the former federal employees and obtaining 
their cooperation in providing data for the third segmeni. (See table 7.2.) 

In terms of scope, an effort such as this could be phased in or all compo- 
nents initiated concurrently. Each approach has both benefits and risks. 
One could, for example, begin the assessment on a more modest scale by 
gathering less data, such as by using only a few indicators of quality. 
However, those indicators that are most accessible, for instance, the col- 
lege entrance test scores stored in computer files at the testing organiza- 
tions, are limited in interpretability; publicizing results on any isolated 
measure, according to many of the people we consulted in doing the 
review, could damage the eventual acceptability of the overall assess- 
ment. Another approach would be to begin either the new employee or 
separated employee studies without comparison groups (to reduce the 
data-gathering load). Wc- see great risks to interpretability if comers 
were cut by eliminating the comparison groups. To omit them invites 



Timing and Phasing of 
the Assessment 
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Table 7.2: Relative Difficulty of Parts of 
the Assessment 



Segment 


Sample 


Degree of Difficulty 


Entering workforce 


Federal 


Not difficult 




Nonfederal (comparison 
group) 


Difficult (establisnrng universe, 
locating sampled respondents) 


Core assessment 


Overall 


Moderately difficult because of 
overall complexity of assessment 
dimensions and double sample 
(emp oyees and supervisors) 


Separating workforce 


Nonfederal 


Moderately difficult (establishing 
universe, locating sampled 
respondents, obtaining cooperation) 




Federal (compariscn group) 


Not difficult 



more of the same speculation (for example, on whether other employers 
are more successful in attraicting quality in competition with the federal 
sector), which is precisely what the overall assessment is designed to 
reduce. 



One way to start smaller might be to begin the core assessment, which 
includes the most comprehensive defmition of quality and the most 
interpretable set of indicators but to reduce the workload by starting 
with fewer selected occupations than the full set of 23 listed in appendix 
II. Those occupations not selected for the initial year's work could be 
assessed for their first time in subsequent years. Trend analyses would 
thus be done on different groups in different years, for example, attor- 
neys in the years 1988 and, if the interval is four years, 1992, while 
nurses could be studied in 1989 and 1993. Each initial occupational sam- 
ple would include some employees in the first years of service, which 
would provide the limited picture of quality of those entering (without 
the important comparison group, as just described), though these results 
would be able to be compared with the fuller picture on the rest of the 
workforce gathered at the same time. Conducting the assessment in this 
way, however, would not allow the msyor current questions to be 
answered and can only be recommended as a contingent measure. 



Conclusions believes a data-collection program such as that outlined in chapters 

4-6, established at congressional direction and with sufficient resources 
to carry out the work in a methodologically sound way, would provide 
useful information. The results could satisfy the strong congressional 
interest m the quality of the federal workforce ard could also aid in 
decisions on personnel policy in the executive branch. 
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An assessment of federal workforce quality employing the definition 
proposed in chapter 2 is feasible, using new data from samples of fed- 
eral employees and others. Three major questions would organize the 
inquiry, corresponding to three parts of a model of the workforce as 
composed of a core — the current workforce — with flows in and out. 
The core assessment would determine changes in the overall quality of 
the workforce over time. The other two assessments would answer ques- 
tions about the quality of those coming into and leaving federal jobs 
based on specific comparisons to other relevant groups. 

Because federal agency data are now not designed for the purpose and 
because no agency has a mandate to do such an assessment, a good deal 
of new work is necessary, which will take time to produce the full antic- 
ipated results. Therefore the organizational and funding arrangements 
are important to consider so that the assessment can count on the 
needed expertise, commitment to the task, independence, and stability. 



Recommendation to 
the Congress 



To help provide generalizable answers to questions about the quality of 
the federal workforce and changes in it over time, we recommend that 
the Congress authorize a continuing program to assess workforce qual- 
ity based on the framework outlined in this report. 



Agency Ck)mments 



We discussed the proposed assessment with officials at the Office of 
Personnel Management and the Merit Systems Protection Board. The 
officials in general raised no major problems with the proposed defini- 
tion of quality, the initial indicators, or the broad feasibility of the 
designs for the three segm-^i^.ts of the assessment. However, in accord- 
ance with the requester s wishes, we did not obJain written conuncnts 
on a draft of this report. 
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August 19, 1986 



Committee on $o<t Office 
anb Ctbil l^erbire 

Mu^nglon* 9€ 20515 



TEUfHONE (202) 225-4054 



The Honorable Charles A. Bowsher 
Comptroller General 
General Accounting Office 
441 G Street, N-W. 
Washington, D . C . 20548 



Dear Mr. Bowsher: 

Eleanor Chelimsky, Director of the Program Evaluation and 
Methodology Division of the General Accounting Office, wrote an 
excellent and provocative letter on March 4, 1986, to Andrew 
Feinstein of the Subcommittee on Civil Service about the 
difficulties of measuring changes in the quality of the Federal 
workforce. Chairwoman Patricia Schroeder of the Civil Service 
Subcommittee sent copies of the cnelimsky letter to academics anO 
practitioners in the field of public administration to garner 
their views on measuring workforce quality. By and large, their 
responses confirmed the difficulties raised in the Chelimsky 
letter. 

In spite of the difficulties inherent in measuring workforce 
qua"" ity , there is a strong need to do so. How wel 1 the 
government provides needed services at a reasonable price 
depends, in no small measure, on the quality of the workforce. 
Moreover, Congr«'*s<5 ^viu. t^e Iresiderif 3lq ;rr€':-u,>ntly asked not to 
take some action because of its effect on workforce quality; yet, 
these policymakers have no way of knowing whether there will be 
an effect and^ if so, how sei ious the effect will be . Indeed, 
while many seem to have an opinion, there is no data about 
whether the quality of the Federal workforce has increased or 
declined in the past decade. 

There is no good method to capture changes in quality in the 
past. We can, however, establish a baseline now from which 
future changes can be measured . Therefore , the Committee 
requests that the General Accounting Office begin developing a 
methodology which could allow for the measure of personnel 
nuality. Clearly, further work in this area could lead to the 
conclusion that such measurement is not possible . If such 
measurement possible, however, it would be of great value. 
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Honorable Charles A. Bowsher 
August 19, 1986 
Page 2 



Specifically, the Committee requests that the General Accounting 
Office determine whether it is feasible to construct a data base 
permitting measurement of both current status and trend data on 
civilian workforce quality and, if it is possible, provide a 
design for the development of such a data base. Ideally, the 
Committee would like a fully developed data base design from the 
General Accounting Office which the Committee could ask the 
Office of Personnel Management to implement. 

The Committee requests that the work be performed by a team of 
General Accounting Office experts, drawn both from the Program 
Evaluation and Methodology Division and from the General 
Government Division. The Committee wants to minimize the risk of 
recommending initiation of a potentially large effort by the 
Office of Personnel Management only to find that key questions of 
methodology and validity, which could have beer, anticipated, are 
unanswered. 

The Committee recognizes that this request will be challenging 
for the General Accounting Office. It will take time and 

creativity . The Committee is wil ling to give the General 
Accounting Office wide flexibility in selecting study approaches. 
Andrew Feinsrein of the Subcommittee on Civil Service will fc , the 
contact person for this effort. Please keep him fully informed 
of your efforts. 

Thank you for your help. 

With kind regards. 

Sincerely yours. 




WILLIAM D. FORD 
Chairman 
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The core assessment described in chapter 5 should examine indica- 
tors of quality in the workforce within a consistent set of occupa- 
tions across time. (Of course the set could change over the long term, 
after at least two rounds of data-collection establish the general 
trends in indicators for any occupation.) This appendix describes an 
approach to selecting the occupations from which employees would 
be sampled, for that part of the assessment only, based on the size of 
the workforce in each occupation. 

Using OPM data for 1985 (the latest year for which statistics on occupa- 
tions are available), we selected all major professional and administra- 
tive occupations paid on the general schedule or its equivalent. We 
defined "major" as any occupation with at least 8,000 employees. (We 
eliminated the 1710 series, occupational and vocational training, 
because the m^ority of employees are not on the general schedule or an 
equivalent pay system, and the 861 series, aerospace engineer, as the 
list already included 4 other kinds of engineers.) 

In addition, we suggest forming two occupational clusters, one of 13 
physical science jobs (such as physicist, chemist, and geologist) and 
another of 12 biological science jobs (such as ecologist, botanist, and 
zoologist). Each of the individual occupational series has fewer than 
8,000 employees, but as the combined groups with similar qualifications 
and position descriptions are sizable (32,508 in the physical science clus- 
ter and 9,878 in the biological science cluster), and as there is continuing 
concern for the quality of scientific personnel in government, we believe 
it is useful to include them. 

These occupations appear to be an adequately comprehensive and bal- 
anced set from which to sample. The total of 435,000 employees 
includes about 53 percent of the overall professional a.id administrative 
workforce. Of the set of 23 occupations (considering J\e scientific clus- 
ters as two occupations) we are proposing, 1 2 are pro/'^ssional and 1 1 
are administrative. The overall proportion of women is 26 percent in 
professional and 34 percent in administrative occupations: this group of 
occupations includes 31 percent women. 

Table II. 1 shows the 23 occupations meeting our criteria. Table II.2 
shows the two sets of scientific occupations that have been combined. 
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Table 11.1: Occupations Selected for 
Quality Aasessment 



Series 


Title Type* 


Total 
emolovees 


Men 


vWlllwll 


105 


Social Insurance Adminisirator 


A 


21.401 


9,626 


1 1.775 


201 


Personnel Management Special st 
(includes Personnel Offirprl 


A 
r\ 


9.471 


4 827 


4,644 


334 




A 

A 


40.122 


27,493 


12.629 


341 


AdrrnnKtrativp Offirpr 


A 
M 


8.460 


3.805 


4.655 


343 


lyiandyciTicni Mnaiysi 


A 

A 


15.694 


8.164 


7.530 


345 


r^ruyiani Mndiybi 


A 

A 


16.162 


8.850 


7.312 


510 


Accountant 


p 

r 


11.198 


8.006 


3.192 


511 


Auditor 


D 


12.435 


10.079 


2.356 


512 


IRS Agent 


P 


14.293 


10,623 


3.670 


560 


Budget Analyst 


A 
M 


11,359 


4.522 


6.837 


602 


Medical Officer'' 


p 


9.600 


7.864 


1.736 


610 


Nurse^ 


P 


39.109 


3.419 


35.690 


801 


General Engineer 


P 


19,569 


18.863 


706 


810 


Civ 1 Engineer 


P 


16,775 


15.847 


928 


830 


Mechanical Engineer 


P 


13,583 


12,927 


656 


855 


Electronics Engineer 




24,033 


22.952 


1.081 


905" 


General Attorney 


~ P 


17.796 


12.962 


4.834 


1102 


Contracting Specialist 


A~ 


27,871 


14.246 


13,625 


1811" ~ 


Cnminal Investigator 


" A 


24 230 


22.303 


1,927 


1910 ~~ 


Quality Assurance Specialist 


A 


16,829 


14.946 


1.883 


2152 


Air Traffic Controller 


A 


22.660 


20.454 


2.206 


Sub-total, individual series 




392,650 


262,778 


129,872 




Biological Researcher 


P 


9,878 


7.748 


2.130 




Physical Scientist 


P" 


32.508 


28.196 


4.322 


Total, all series 




435,036 


298,712 


136,324 


Source Office of Personnel Management. Federal Civili?.i Workforce Statistics Occupations of Federal 
White-Collar and Blue Collar Workers (Washington, p c October 31 I9fi5l 



"Indicates the type of job showing for each series the 0PM classification as either professional (P) or 
administrative (A) Professional series have positive educational requirements for hinng, while adminis- 
trative series do not 

•These occupations are paid on a pay schedule equivalent to the general schedule 
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Table 11.2: Additional Occupations to Bo 
C/ouped for Quality Assessment 



Physical Scien*'st 


1301 


General Physic.^ Science 


5,323 


4.742 


581 


1306 


He-^lth Physics 


599 


518 


81 


1310 


Physics 


4.278 


4,073 


205 


1313 


Geophyt'cs 


607 


552 


55 


1315 


Hydrology 


2,249 


2,049 


200 


1320 


Chemistry 


7,602 


5,992 


1.610 


1321 


Metallurgy 


444 


426 


18 


1330 


Astronomy and Space Science 


543 


505 


38 


1340 


Meteorology 


2.133 


2.017 


116 


1350 


Geology 


2.601 


2,223 


378 


1360 


Oceanography 


761 


679 


82 


1370 


Cartography 


4,985 


4,077 


908 


1372 


Geodesy 


383 


333 


50 


Total 




32,508 


26,186 


4,322 



Source 0PM (same as table I1 1) Data are as of Octot^er 31 1985 



Series 


Title 


Total 
employees 


Men 


IMNiien 


Biological Researchei 


401 


General Biological Scientist 


4,752 


3.778 


974 


403 


Microbiology 


1,848 


1.166 


682 


405 


Pharmacology 


439 


338 


101 


408 


Ecology 


325 


272 


53 


410 


Zoology 


120 


99 


21 


413 


Physiology 


451 


367 


84 


414 


Entomology 


745 


706 


39 


430 


Botany 


173 


119 


54 


434 


Plant Pathology 


319 


287 


32 


435 


Plant Physiology 


307 


273 


34 


437 


Horticulture 


117 


96 


21 


440 


Genetics 


282 


247 


35 



Total 
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To project costs for the proposed assessments, sample size is a major 
determinant. When precise characteristics of measuring instruments are 
not yet determined, as in the case of the proix)sed employee and super- 
visor surveys to be used in the three segments of the design described in 
chapters 4-6, estimates of sample size must rely on assumptions. The 
other m^or parameter affecting sample sizes is the degree of confidence 
in the results that is desired. The greater the confidence or precision 
desired, the large? che sample must be. 

To estimate the initial sample sizes, we have used conservative assump- 
tions about the characteristics of the instruments and the most stringent 
degree of confidence (that the sample estimate be wrong in either direc- 
tion by no more than 5 percentage points). These planning assumptions 
rer>ult in a needed sample of 400 for each group on which conclusions 
are desired.* 



The assessment of the entering workforce described in chapter 4 
involves only one level of experience (all those surveyed are newcom- 
ers) and we propose that fewer occupations be studied than the 23 nomi- 
nated for the core assessment. Thus the needed samples a* j relatively 
small. The 400 new hires needed for analysis of an occupation should be 
split between federal and nonfederal employees.- 

The total sample size for this segment depends on the number of occupa- 
tions to be examined. Assuming that interest in new hires is not as 
intense for all 23 of the occupations in the core assessment, we project a 
total group of occupations half as lar, or 12 total. Assuming also that 
analysis will be limited to the examination of differences, 12 occupa- 
tions X 400 gives a sample of 4,800. Half "^ou!d be federal employees 
and half nonfederal. 



Sampling the Current The overall professional and administrative workforce should be 

Workforce assessed using a stable and relatively comprehensive set of occupations, 

as outhned in chapter 5 and appendix II. Sampling 400 employees at 



'The effect of relaxing confidence requirements ls dramatic If the tolerable margin of error is 8 
percentage points the required sample drops to 160, with an acceptable margin of 10 percent, the 
sample can be 100 

-This allows acceptable estimates of differences between federal and non-federal employees m each 
occupation If it was desirable to make separate estimates on the survey vanables for each group, 
then the sample size would again be 400 per group per occupation, effectively doubling the sample 
size 

8:.: 



Sample Sizes 



Sampling the Entering 
Workforce 
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each of three levels of experience within each of 23 occupations will 
result in an employee survey sample totalling 27,600. Each sampled 
employee's supervisor is also surv( ed. Not all employees and supervi- 
sors would be surveyed at one time, however, if the assessment plans 
assign different jobs to different four-year intervals. 



Sampling the Exiting For studying separated employees, a sample of 300 (matched to 300 

Workforce stayed in government), or a total of 600, is needed per occupation. 

The number of sampling units, 300 pairs of employees in this case, is 
smaller than in the previous discussion where 400 units (individuals) 
are needed in an occupation for the most careful estimates. Fewer sam- 
pling units are needed since each separated employee is matched with 
another nonseparated employee on several dimensions, and the overall 
variance will therefore be somewhat less. With smaller expected vari- 
ance the number of units sampled can be reduced and still attain the 
same effective degree of precision. The pairs will be matched on age, 
sex, and years of government experience, so formal stratification by 
experience is not needed in the sampling. Total sample size again 
depends on the number of occupations to be examined. The quality of 
people leaving is critical in some jobs but probably not in all of the 23 
suggested for the core assessment, so we have once again estimated the 
total sample size using 12 occupations. No supervisor data are to be 
gathered on this group. For 12 occupations x 600, the sample size would 
be 7,200. 



Pinpointing quality differences by agency in any of the three segments 
of the assessment would require adding another stratification scheme to 
the sampling design. It is not likely that there would be enough people in 
a range of agencies in the basic samples for statistically sound compari- 
sons. The planning would need to consider the fact that some jobs are 
highly concentrated in a few agencies, while others are dispersed. For 
jobs in which a large number of the workforce are employed at a few 
agencies, it would be especially feasible to sample employees in each job 
by agency with confidence that adequate numbers are there for analy- 
sis. For example, for 1 1 of the 23 jobs we propose for assessment 
(shown in table II. 1 in appendix II), 66 percent or more of the workforce 
in each of the jobs can be found in three or fewer agencies. (For six of 
the 1 1 job series, the concentrations are in different sets of agencies. The 
other five include three for which the three military services account for 
66 percent or more ol the workforce, and medical officers and nurses 
which are both concen trated at the va, hhs, and Army.) Meaningful 



Sampling to Obtain 
Agency-Level Comparisons 
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agency comparisons would appear to be possible in those cases. For jobs 
that are more dispersed across numerous agencies, and for which there 
may be smaller numbers at each agency, acceptable agency-level sam- 
ples may not be possible. There appear to be some special problems in 
attempting agency-level analysis in the study of entering employees, 
also.-^ Further technical development should consider the trade-offs 
among feasibility, cost, and interpretability for various sampling 
approaches. 

Table III. 1 summarizes the initial samples for each segment of the 
assessment, without provision for agency-level analyses. 



Table Sample Sizes Needed for 
Three Segments of the Proposed 
Assessment of Federal Workforce 
Quality 



Assessment segment 


Number of occupations 


Other strata 


Total group 


Entering employees 








Federal employees 




None 


2.400 


Nonfederal employees 


12 


None 


2.400 


Current employees 








Employees 




3 


27,600 


Supervisors 


c 


None 


27,600 


Exiting employees 








Separated employees 


12^ 


None 


3,600 


Current ennployees 


12d 


None 


3,600 



^Number of occupations for estimating purposes only, actual number could be larger or smaller, 
depending on fina' 'osign of the assessment 

^Recommended number of occupations, see appendix II 

^Supervisors would not be sampled, but would be identified as the supervisors of record for the sam- 
pled employees 

^Within each job series selected for study, current employees would be chosen by matching those 
leaving on sex, age, and experience 



Ck)St EstimSltC hasic cost elements of the proposed assessment program are the 

staff time for development and analysis and the contract costs of survey 
administration and data preparation. Large-scale surveys are sometimes 
done by government employees (such as those in the Census Bureau) but 
are more commonly contracted to specialized private-sector firms. The 
core assessment we estimate would involve up to four staff years of 
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^Relatively small numbers may be hir^ m a single occupation in specific agencies in one year; solving 
this problem by drp ving a survey sample from an aggregated group of several years' new hires raises 
other problems. For example, employees sampled from CPDF records showing them to be working in 
a particulpr agency may, by the time they actually complete a survey instrument, have changed 
agency And if the span for agency-level sampling has to be as broad as three years to find needed 
numbers of new people in a job series, in that period there may have been too many changes in policy 
and work conditions to permit the desired clear-cut analyses of effects on the quality of people 
attracted. 
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time for development and pretesting and for analysis after the data are 
received in usable form. The entry and exit segments of the design 
wor'd not add appreciably to the staff effort needed in instrument 
development, but as they present some unique sampling and analysis 
challenges, we hav^ added another half staff-year. For per-unit survey 
processing costs, w;^ have used the average cost experienc by ivlpl in 
a r^ent contract for processing its Merit Principles Survey, which went 
to ^oout 21,000 ff deral employees in a manner similar to that we are 
proposing, mspb officials timated that they paid a contractor $1.50 per 
irvey for printing and distributing the surveys, managing their receipt, 
entering the data into computers (using survey response foniiS that can 
be read automatically by optical scanning equipment), and producing a 
usable data file for analysis. 

Ck)sts of drawing various samples of federal employees are simply the 
computer and staff costs for the opm data center, as the samples can all 
be di awn from opm's Central Personnel Data File. This is the case for all 
samples in the proposed program except the comparison gi : . 
nonfederal new ^ires, which ^'"esents special problems. 



As discussed in chapter 4, there are significantly different options for 
locating the necessary comparison group of nonfederal new hi^'es for 
that segment of t le assessment. In one method, people new to their jobs 
could be found by asking a sample of employers to identify those they 
have hired in specific occupations within a specified time period. In a 
second method, a group of recent college graduates, who by definition 
will be new to any job, could be traced starting with samples from 
school records. 

The approach through employers has the advantage of greater coverage 
of the new hire group, including both inexperienced and experienced 
individuals new to heir jobs. It has the disadvantage oi uncertain but 
potentially significant costs and burdens, as the state and federal agen- 
des who maintain employer lists do not have experience surveying 
employers (or working through them) for the particular purpose of 
obtaining information on individuals. The Bureau of Labor Statistics 
re orted on their experience gathering routine data series on employ- 
ment and the approach the states use to reach samples of ^^rms. Bureau 
officials said they have also used the same approach for oue-time stud- 
ies of national population^* of employers, such as a recent survey on 
employee drug-use. But all the data gathered using the employer sain- 
pling franfies are summary statistics on groups of people. Because the 
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costs of many activities in che joint BLS-state oiatistical reporting on 
occupations are included in cost-sharing agreements negotiated with the 
states, BLS could not estimate the specific costs of adding our particular 
requirements to existing agreements.^ bis officials did not recommend 
working through states, but concentrated on options based on drawing 
national samples of firms likely to have new hires in the occupations of 
interest, using bls merged files of employers drawn from state records. 
Even so, because the process would involve several steps beyond any 
they have attempted \'nth employers, they could not be sure that the 
survey effort would be feasible or what it would cost. We explored three 
alternatives: employers could supply information about individuals' 
back^ounds, drawn from files, though we doubt this data source would 
be comprehensive, as discussed in chapter 4; employers could distribute 
surveys to employees to be returned to the assessment (an approach 
similar to that to be used with federal employees); or employers could 
provide addresses which ♦;he assessment agency or bis could use to mail 
surveys. If an employer had hired large numl ^rs of people in the target 
occupations, there would need to be a method for drawing a sub-sample 
within that group, to lessen burden. 

The feasibility of the second approach, finding new hires by tracing 
recent college graduates, is shown by surveys done by the Department 
of Education. The department periodically surveys a national sample of 
people who have just received bacheloi s' and masters' degrees, about 
one year after their degree date. The survey does locate some who have 
taken fer' aral jobs (572 federal <»mployees responded to the last depart- 
ment survey, along with ahout iv;,000 others), and thus appeared to 
include the two groups needed tor this assessment within its design. But 

^ )und that the group of federal employees is too diverse and the 
nui /ers of people in any particu'^r federal occupation too small to per- 
mit analysis of ^ jality indicators within specific professional and 
administrative occupational seri^^, which is our design requir^^ment 
Attempting to augment the sample to insure enough federal employees 



^As an example, BLS officials told us tnat a federal agency ^hat needs detailed data on scientists and 
engineers transfers 1240,000 per year to augment dau-jjathering on thos occi .pations alone. 
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(about 200 are needed in each s jiected occupation) raises questions of 
feasibility and cost.*^ 

Thus the greatest potential use of the department's survey would be to 
provide the comparison group data only. That is, entering federal 
employees could be surveyed directly using a sample drawn from opm 
files, while data on those in some similar nonfederal jobs could be drawn 
from those sampled in the Department of Er^ncation survey. (Chapter 4 
discusses limitations of this comparison groui ) We reviewed the depart- 
ment's data from the most recent survey to see if there were the 200 
nonfederal employees needed for analysis in the 23 occupational series 
we selected. Department officials told us only three jobs came near that 
target. (Their responses included 899 computer specialists, 452 account- 
ants, and 168 electncal engineers.) All the other series we asked about 
had fewer respondents. We explored the feasibility of locating larger 
numbers in the spr cif ic needed occupations. Schools provide the Depart- 
r ent with lists of graduates to use in sampling. The lists also show each 
graduate's mi^jor field of study. Past surveys can be used to estimate the 
association between field of study and first job which can be used to 
guide oversampling in future studies, so that, for example, a certain 
number of additional .nath, science, or engineering majors can be sam- 
pled to obtain an estimated number of additional respondents who are 
scientists and engineers. 

After we completed our data-gathering, department officials told us 
they decided to discontinue the specific Recent College Graduates Sur- 
vey and to establish in 1990 a new longitudinal study involving surveys 
of students while in school and also 1, 3, and 6 years after graduation. 
The feasibility and cost of using the new survey as a source of the 
nonfederal comparison group can be explored when planning for the 
1990 survey begins. Specific issues would include the need for questions 
on educational background and academic achievement, work experience 
if any, and specific occupation; and the sample would need to yield the 
needed 200 in each of the selected occupations of interest. 

Tab. . III.2 sunm>^rizes the cost elements discus.« jd above. The display 
includes only core professional staff; thus it does not include costs of 



''For example, OPM could provide a list of ne\/ federal employees in the selected occupations, which 
could be matched with the student lists provided by colleges during the Education Department survey 
sampling stage, and those found in this way could be oversampled in an attempt to mcrease the 
fv»deral employ e group for comparative analysis. The department docs some automated and hand 
checking of the. student lists already as part of sampling, for instance to insure over-samplmg of 
Hispanic-sumamed individuals. But as only half the 400 schools mvolved have automated student 
records, the full matching process, both by computer and by hand, could be costly> 
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supervision of the office that is assigned the work or support staff in 
such an office that would be needed for study tasks and report produc- 
tion. It also does not include cost estimates for either alternative 
approach we have considered to sampling newly-hired nonfederal 
employees (reaching them either through employer lists kept at bus or 
state offices, or through a plamied Department of Education survey of 
recent graduates sampled from school records). (See table IIL2.) 



Table 111.2: Estimated Coats of the 
Proposed Assessment of Federal 
Workforce Quality* 



Cost Element 


Cost 


Survey design, development, pre-testirg, and analysis of data (maximum of 




4 5 staff years at $472 per day) 


424.800 


Sampling (OPM interna! charges) 


10,000 


Survey administration* (64,800 surveys at $1 50)^ 


97,200 


Computer costs for analysis (12 months at $3000) 


36,000 


Estimated contract costs for computer matching on 4,800 new hires at 




college entrance ifsX firms 


50.000 


Report production^ . 


100.000 


Total "f^ 


$718,000 



^Assumes starting aH segments of the assessment and completing one baseline cycle of data-gathering 
and analysis on enjenng, current, and exiting employees, with comparison groups as discussed in the 
text Does not includfe costs of overall supervision and management, or indirect costs such as space 
rental, as these could vary widely under different arrangements for conducting the assessment Also 
does not inclu'le estimated cost of edh^ ontion for obtaining survey data on comparison group of 2400 
newly-hired nonfederal employees 

^Assumes separated employees can be surveyed at the same cost as current employees, since 
addresses are kept on file at agencies for several years 



There is an indirect cost to surveys, in the salary paid for the time spent 
completing the survey, time in which other work is put aside. For the 
current federal employees proposed to be surveyed in the three parts of 
the assessment, the tot.l value of the time spent in ' he surveys is esti- 
mated to be $766,ld7J' (In addition, 2,400 people recently hired by 
nonfederal employers would b • surveyed, and also 3,600 former federal 
employees. We did not attempt to estimate the value of their time spent 
on the survey.) These costs, which are salary dollars that 'vill be spent 
anyway, are not shown in table IIL2, to avoid confusion wi>^h direct dol- 
lar costs that would require additional federal expense. 



'*The fij?uie assumes that any su-vey for this assessment would require no more than 45 minutes of 
• duty time to complete Each sample would contain about half administrative and half professional 

employees. To calculate the sai.jn/ cost of the survey, the new-hire sample was assumed to be evenly 
livided between grades GS-7 ana GS-9 (though some unknown fraction of the actual sample would be 
experienced people hired at higher grades). The overall wwxforce sample was assumed to be evenly 
divided between grades GS-1 1 and GS-12, which are the average grades for administrative and pro- 
fessional occupations. The group of current cmobyees to be matched with those who separated was 
assumed to be evenly divided between grades OS- 12 and OS- 14. 
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We chose an advisory group for the project consisting of 1 1 members 
with expertise in economics, educational and psychological measure- 
ment, political science, and evaluation design, as well as experience in 
Ck>ngress, in m^yor business corporations, in federal personnel manage- 
ment and in feu ral employee groups, at the state level, and in academia. 
The advisory panel reviewed the project design ana approach at the 
beginning. The group met on January 29, 1988, when we presented ten- 
tative results of the work so far and reviewed the basic feasibility t)f an 
assessment using samples. Advisory froup membei's completed their 
work by making comments on a draft of our report. The panel members 
weie: 

Dr. Joseph L. Fisher 
Special Assistant to the Pr*, dent 
George Mason University 
Fairfax, VA 

Dr. Eli Ginzberg 

Ck)nservation of Human Resources 
Ck)lumbia University 
New York, NY 

Mr. Jerry Klepner 
Director of Legislation 

American Federation of State, County and Municipal Employees 
Washington, DC 

Mr. Paul O'NeiU 
Chairman of the Board 
Aluminum Comp^iny of America 
Pittsburgh, PA 

Professor Nelson Polsby 
Department of Political Science 
University of California 
Berkeley, GA 

Mr. J. M. Schulman 
Director of Personnel 
U.S. Department of Energy 
V/ashington, DC 

97 
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Professor Lee Sechrest 
Department of Psychology 
University of Arizona 
Tucson, AZ 

Dr. Marshall Smith 
Dean, School of Education 
Stanford University 
Stanford, CA 

Mr. Ray Sumser 

Director of Civilian Personnel 

U.S. Army 

Washington, DC 

Dr. Mary Tenopyr 

Selection and Testing Director 

AT&T 

Short Hills, NJ 

^ofessor Carol Weiss 
Graduate School of Education 
Harvard University 
Cambridge, MA 



In addition, the development of our approach has benefited from the 
advice and review of a number of groups and Individuals in addition to 
the advisory panel. We arranged group consultation conferences with 
three associations in the field: 

• American Society for Personnel Administration 

• American Society for Public Administration 

• International Personnel Management Association, Federal Section 

We addressed questions about the issues in assessing the workforce to 
the heads of four federal employee groups and held individual inter- 
views with staff of each group. These groups were: 

• American Federation of Government Emplovees 

• Federal Managers Association 

• National Federation of Federal Employees 

• National Treasury Employees Union 
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We met with other groups to present the project and obtain views, 
including the Panel on Public Service of the National Academy of Public 
Administration and the Human Resources Task Force of the President's 
Council on Management Improvement. 

In addition, we discussed the problem of workforce quality assessment 
and various solutions with individuals in several organizations involved 
with the issues in both public and private sectors. Individuals who made 
helpful contributions include: 

Mark Abramson 

Center for Excellence in Government 
Sue Berryman 

National Center on Education and Employment 
Richard Bums 

Center for Occupational and Professional Assessment, Educational Test- 
ing Ser/ice 

Dave Crockett 

American College Testing Program 

Marvin Du*inette 
University of Minnesota 

Edie Goldenberg 
University of Michigan 

Glenn Gotz 

The Rand Corp., Defense Manpower Research Center 
Richard Hackman 

Harvard Graduate School of Business 

Bemd Hasenkantp 
Educational Testing Service 

William Kennish 

Law School Admissions Council 

Steve Kerr 

Universi* of Southern California C " 
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Allen I. Kraut 
IBM 

Edward Lawler 

University of Southern California 
John Lee 

National Center for Postsecondary Governance and Finance 
Charles Levine 

National Commission on the Public Service 

Michael Uebman 
41cMani$ Assodates 

ICchard Mansfield 
McBer and Company, Inc. 

Eugene McGregor, Jr. 
Indiana University 

Brian Morgan 

Opinion Research Corporation 

James Perry 
Indiana (Jnive^^ity 

Karlene Roberts 

University of California, Berkeley 
Robert Weatherall 

Massachusetts Institute of Technology 

Alexandra Wigdor 
National Research Council 
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In 1986, the Chair of the House Subcommittee on Civil Service circulated 
for comment an earlier informal gao communication on assessing quality 
in the workforce. Responses from 12 federal agencies as well as many 
individuals helped shape the plans for the work presented in this report. 
Some individuals already cited above wrote in response and also subse- 
quently provided additional help; others who sent the committee infor- 
mation and views on ways to assess quality in the workforce were: 

Carolyn Ban 

State University of NewTork at Albany 

Japies Bowman 
Florida State University 

Colin Campbell 
GeorgetowTi University 

William Carey 

American Association for the Advancement of Science 

James Fesler 
Yah University 

Gregory Gaertner 
WESTATCorp. 

Michael Hansen 
American University 

Sar Levitan 

George Washington University 
Paul Lorent7en 

University of Southern California 
Bradley Patterson 

American Society for Public Administration 

Barbara Romzek 
University of Kansas 

Bernard Rosen 
American University 



Earlier 

Correspondence With 
the Committee 
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Richard Schmidt 

Scanlon, Hastings and Schmidt 

David Stanley 
Vienna, VA 

James Sundquist 
Brookings Institution 

Frederick Thayer 
University of Pittsburgh 

FYank Yeager 
EDA Systems 
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Requests for copies of gag reports should be sent to: 

U.S. General Accounting Office 
Post Office Box 6015 
Gaithersburg, Maryland 20877 

Telephone 202-275-6241 

The first five copies of each report are free. Additiorial copies are 
$2.00 each. 

There is a 25% discount on orders for 100 or more copies mailed to a 
single address. 

Orders must be prepaid by cash or by check or money order made out to 
the Superintendent of Documents. 



